KEX-bench

Evaluating Coding Agents on Kernel Exploit Generation

EMNLP 2026 Main (Oral Presentation)

KEX-bench evaluates coding agents on exploit primitive generation against real Linux and Windows kernels. The benchmark contains 45 task instances across 40 CVEs, with deterministic, primitive-specific verification.

Linux includes 25 tasks across 20 CVEs: 9 kernel address leaks, 6 instruction-pointer control tasks, 6 heap reads, and 4 heap writes.

#	Model	Success	Linux	Windows	Provider
1	Opus 4.6Claude Code	68.9% 31/45 LinuxWindows	23/25	8/20	Anthropic
2	Sonnet 4.6Claude Code	53.3% 24/45 LinuxWindows	19/25	5/20	Anthropic
3	GPT-5.4Codex	42.2% 19/45 LinuxWindows	12/25	7/20	OpenAI
4	Sonnet 4.6OpenCode	35.6% 16/45 LinuxWindows	10/25	6/20	Anthropic
5	GPT-5.3-CodexCodex	24.4% 11/45 LinuxWindows	6/25	5/20	OpenAI
6	GPT-5.4 MiniCodex	11.1% 5/45 LinuxWindows	3/25	2/20	OpenAI
7	Haiku 4.5OpenCode	4.4% 2/45 LinuxWindows	2/25	0/20	Anthropic
8	GPT-5.4 MiniOpenCode	4.4% 2/45 LinuxWindows	2/25	0/20	OpenAI
9	Kimi K2.5OpenCode Open	4.4% 2/45 LinuxWindows	2/25	0/20	Moonshot AI
10	Gemma 4 31BOpenCode Open	2.2% 1/45 LinuxWindows	1/25	0/20	Google
11	Haiku 4.5Claude Code	0.0% 0/45 LinuxWindows	0/25	0/20	Anthropic

Results use the released 45-task evaluation suite. Each agent/model pair is evaluated once per task in each setting.

#	Model	Success	Tokens	Runtime	Provider
1	Opus 4.6Claude Code	92.0% 23/25	113.7M	22.5h	Anthropic
2	Sonnet 4.6Claude Code	76.0% 19/25	176.5M	30.3h	Anthropic
3	GPT-5.4Codex	48.0% 12/25	123.7M	9.9h	OpenAI
4	Sonnet 4.6OpenCode	40.0% 10/25	449.9M	21.4h	Anthropic
5	GPT-5.3-CodexCodex	24.0% 6/25	56.7M	7.0h	OpenAI
6	GPT-5.4 MiniCodex	12.0% 3/25	87.2M	5.7h	OpenAI
7	Haiku 4.5OpenCode	8.0% 2/25	41.8M	3.7h	Anthropic
8	GPT-5.4 MiniOpenCode	8.0% 2/25	84.2M	5.7h	OpenAI
9	Kimi K2.5OpenCode Open	8.0% 2/25	90.9M	13.0h	Moonshot AI
10	Gemma 4 31BOpenCode Open	4.0% 1/25	14.4M	11.2h	Google
11	Haiku 4.5Claude Code	0.0% 0/25	39.6M	3.7h	Anthropic

Results use the released 45-task evaluation suite. Each agent/model pair is evaluated once per task in each setting.

#	Model	Success	Tokens	Runtime	Provider
1	Opus 4.6Claude Code	40.0% 8/20	402.9M	32.9h	Anthropic
2	GPT-5.4Codex	35.0% 7/20	288.2M	21.8h	OpenAI
3	Sonnet 4.6OpenCode	30.0% 6/20	614.6M	26.2h	Anthropic
4	Sonnet 4.6Claude Code	25.0% 5/20	399.3M	38.5h	Anthropic
5	GPT-5.3-CodexCodex	25.0% 5/20	209.8M	14.3h	OpenAI
6	GPT-5.4 MiniCodex	10.0% 2/20	171.6M	7.0h	OpenAI
7	Haiku 4.5OpenCode	0.0% 0/20	28.8M	2.6h	Anthropic
8	GPT-5.4 MiniOpenCode	0.0% 0/20	7.7M	1.4h	OpenAI
9	Kimi K2.5OpenCode Open	0.0% 0/20	43.1M	7.1h	Moonshot AI
10	Gemma 4 31BOpenCode Open	0.0% 0/20	12.9M	18.8h	Google
11	Haiku 4.5Claude Code	0.0% 0/20	59.7M	7.8h	Anthropic

Results use the released 45-task evaluation suite. Each agent/model pair is evaluated once per task in each setting.

News

EMNLP 2026 Main: “Evaluating Coding Agents on Kernel Exploit Generation” has been accepted for an oral presentation.

Overview

KEX-bench measures whether coding agents can construct exploit primitives against real operating-system kernels. Each task runs in an isolated virtual machine, exposes controlled analysis tools, and uses a deterministic verifier to check primitive-specific success.

Evaluation Settings

with PoC: the agent receives a reference trigger PoC as a starting point.
without PoC: the agent receives no trigger PoC.

Both settings require verifier-confirmed primitives under the official tool-call budgets. Crashes and textual claims do not count as solved tasks.

Scoring

Scores are pass@1: solved tasks divided by the full task count for the selected platform. Overall uses all 45 task instances; Linux uses 25; Windows uses 20.

Select a model to view its platform runs, primitive breakdowns, token totals, runtime, and tool-call counts.