Benchmarks / AIDev
AIDev
Built by Queen's University · Hao Li, Haoxiang Zhang, Ahmed E. Hassan · released 20 Jul 2025
A public dataset of pull requests that autonomous coding agents (OpenAI Codex, Devin, GitHub Copilot, Cursor, Claude Code) opened on GitHub, with review timelines, code changes, and merge outcomes. The paper compares agent PRs with human PRs in popular repositories.
Frontier
65.3%
PR acceptance rate (AIDev-pop)
OpenAI Codex
20 Jul 2025 · Source: Queen's University (benchmark maintainers)
Table 5, AIDev-pop. 21.3% rejected. Accepted Codex PRs closed in a median 0.3 hours versus 3.9 hours for human PRs. Human baseline 76.8% accepted. Body text gives 64% from Figure 3.
No fixed reference point (for example Elo scores or field signals).
What it measures
How agent-authored pull requests fare in real open-source review. The headline is the acceptance rate: the share of PRs in the AIDev-pop subset (popular repositories) that were merged, per agent, against a human baseline from the same repositories. The paper also reports task type mix, review turnaround time, and who reviews the PRs. The paper version covers 456,535 agent PRs across 61,453 repositories; the refreshed dataset (to 2025-08-01) has 932,791 PRs.
PR acceptance rate (AIDev-pop): Share of an agent's pull requests in popular repositories that were merged, from Table 5 of the paper. Higher is better.
No fixed reference point, so we do not rate its status. Field signal from observational data. The human baseline of 76.8% is shown for context, not as a parity line.
Facts
- Grain
- Task level: bounded tasks with a clear spec
- Environment
- field-data
- Grading
- field-signal
- Human reference
- Human-authored PRs in the same AIDev-pop repositories: 76.8% accepted, 16.1% rejected (Table 5).
- Reuse
- Not checked in this session. Dataset is on Hugging Face and Zenodo (DOI 10.5281/zenodo.16919272). (unknown)
Limits to keep in mind
- Observational data. Agents are used by different people on different repositories, so acceptance rates mix agent quality with user selection and repository norms. Source
- Agents are found by GitHub search patterns (for example head:codex/, Co-Authored-By: Claude), which miss PRs that do not carry the pattern. Cursor and Claude Code are mostly used locally, so their GitHub footprint is small. Source
- The paper's numbers were captured about two months after several agents launched (Codex started 2025-05-16). The authors say the gaps show how early the transition is. Source
- The paper defines AIDev-pop as repositories with more than 500 stars; the repo README says more than 100 stars and gives different PR counts. We use the paper's Table 5. Source
Recorded results
A selection that shows the frontier over time. The full leaderboard is at the source.
| System | PR acceptance rate (AIDev-pop) | Date | Source |
|---|---|---|---|
| OpenAI Codex · frontier | 65.3% | 20 Jul 2025 | Queen's University · primary |
| Claude Code | 52.5% | 20 Jul 2025 | Queen's University · primary |
| Cursor | 51.4% | 20 Jul 2025 | Queen's University · primary |
| Devin | 48.9% | 20 Jul 2025 | Queen's University · primary |
| GitHub Copilot | 38.2% | 20 Jul 2025 | Queen's University · primary |
Timeline
- 20 Jul 2025 — AIDev dataset of agent pull requests released. Source
Where it sits in the atlas
Go to the source
Last checked 23 Sep 2026 against 3 primary sources. See an error? Tell us.
How to cite
Credit the original work first: AIDev by Queen's University (https://arxiv.org/abs/2507.15003).
Then, if you used this page:
Can Agents Work. "AIDev: frontier results and sources." https://canagentswork.com/benchmarks/aidev-dataset/ (accessed 2026-09-24). CC BY 4.0.@misc{caw-aidev-dataset,
title = {{AIDev: frontier results and sources}},
author = {{Can Agents Work}},
year = {2026},
howpublished = {\url{https://canagentswork.com/benchmarks/aidev-dataset/}},
note = {Accessed 2026-09-24. CC BY 4.0}
}