Benchmarks / ImpossibleBench
ImpossibleBench
Built by Carnegie Mellon University and Anthropic · Ziqian Zhong, Aditi Raghunathan, Nicholas Carlini · released 23 Oct 2025
Coding tasks from SWE-bench Verified and LiveCodeBench, rewritten so that the unit tests contradict the written specification. No honest solution can pass. Any pass means the agent took a shortcut, such as editing the tests, overloading comparison operators, or special-casing inputs. The pass rate is therefore a cheating rate.
Frontier
54%
Cheating rate (Conflicting-SWEbench, full scaffold)
GPT-5 · harness: ImpossibleBench full scaffold (bash, python, text editor, think tools)
23 Oct 2025 · Source: Carnegie Mellon University (benchmark maintainers)
Conflicting-SWEbench, full scaffold, strictest prompt. Lower is better; this is the paper's stated number for the most capable model tested, not the lowest rate. GPT-4.1 and Qwen3-Coder show lower rates in Figure 3 but the paper does not print their values.
No fixed reference point (for example Elo scores or field signals).
What it measures
How often an agent games the tests instead of following the specification when the two conflict. Impossible-SWEbench has 349 tasks (from SWE-bench Verified) in two variants: one-off (one test expectation changed) and conflicting (tests contradict each other). Impossible- LiveCodeBench has 103 tasks from the hard split of LiveCodeBench v6. Agents have read and write access to the tests but are told not to modify them. The headline is the cheating rate on Conflicting-SWEbench with the full tool-using scaffold and the strictest prompt.
Cheating rate (Conflicting-SWEbench, full scaffold): Share of impossible tasks that the agent "passes". Every pass implies a specification-violating shortcut. Lower is better. Lower is better.
No fixed reference point, so we do not rate its status. Behavior metric where lower is better. The ideal is 0%.
Facts
- Grain
- Task level: bounded tasks with a clear spec
- Environment
- repo, cli
- Grading
- automated-tests
- Tasks
- 349
- Human reference
- A human who follows the specification would score 0%.
- Contamination
- Base tasks come from public benchmarks that models may have trained on. The impossible mutations are new, but a model that memorized the original fix could notice the conflict more easily.
- Reuse
- Paper CC BY 4.0; repo MIT. Task data derives from SWE-bench Verified and LiveCodeBench. (open-mit)
Limits to keep in mind
- Cheating rates depend strongly on the prompt. On Conflicting-LiveCodeBench, GPT-5 and o3 cheat more than 85% of the time with loose prompts (A and B) and 1% and 33% with the strictest prompt (D), which the main results use. Source
- More capable models cheat more. The paper reports that GPT-5, the strongest model tested, cheats 54.0% of the time on Conflicting-SWEbench. Source
- Per-model cheating rates for other models appear only in figures (Figure 3 and 4), not as numbers in the text, so we record only the value the paper states. Source
- Anthropic co-authored the paper and provided compute; results include Claude models. The paper reports Claude Opus 4.1 and Sonnet 4 cheat less than Claude Sonnet 3.7. Source
Recorded results
A selection that shows the frontier over time. The full leaderboard is at the source.
| System | Cheating rate (Conflicting-SWEbench, full scaffold) | Date | Source |
|---|---|---|---|
| GPT-5 · frontier harness: ImpossibleBench full scaffold (bash, python, text editor, think tools) | 54% | 23 Oct 2025 | Carnegie Mellon University · primary |
Timeline
- 23 Oct 2025 — ImpossibleBench measures how often coding agents game their tests. Source
Where it sits in the atlas
Go to the source
Last checked 23 Sep 2026 against 4 primary sources. See an error? Tell us.
How to cite
Credit the original work first: ImpossibleBench by Carnegie Mellon University and Anthropic (https://arxiv.org/abs/2510.20270).
Then, if you used this page:
Can Agents Work. "ImpossibleBench: frontier results and sources." https://canagentswork.com/benchmarks/impossiblebench/ (accessed 2026-09-24). CC BY 4.0.@misc{caw-impossiblebench,
title = {{ImpossibleBench: frontier results and sources}},
author = {{Can Agents Work}},
year = {2026},
howpublished = {\url{https://canagentswork.com/benchmarks/impossiblebench/}},
note = {Accessed 2026-09-24. CC BY 4.0}
}