Can Agents Work?

Benchmarks / SOP-Bench

SOP-Bench

Built by Amazon · Subhrangshu Nandi, Arghya Datta, Rohith Nama, Udita Patel, et al. · released 9 Jun 2025

Amazon's benchmark of whether an agent can carry out a written standard operating procedure the way a trained operator would. Domain experts wrote 12 procedures (13 with an easy and a hard variant), for example patient intake, dangerous-goods classification, know-your-business checks, customer service, content flagging, and warehouse package inspection. Each comes with mock tools and test cases with known answers.

Frontier

72.4%

Task success rate

Claude 4 Opus (ReAct agent) · agent: SOP-Bench ReAct baseline

23 Feb 2026 · Source: Amazon (benchmark maintainers)

Best result in the paper (abstract and Table 6, 0.724), mean over 13 procedures. Table 4 shows 0.72 with a standard error of 0.08; the text of section 5.1.2 says 73.8%. See data/conflicts/sop-bench-claude-4-opus-react-tsr.yaml. Per procedure the ReAct run ranges from 27% (content flagging) to 100% (patient intake).

Strong

The best result is at 60–90% of the ceiling, or at or above human parity.

SOP-Bench: Task success rate over time, 3 recorded results. 0%20%40%60%80%100%Feb 2026Feb 2026Feb 2026Mar 2026Mar 2026 Claude 4 Opus (FC agent): 68.2% (23 Feb 2026) Claude 4.5 Sonnet (ReAct agent): 63.3% (23 Feb 2026) Claude 4 Opus (ReAct agent): 72.4% (23 Feb 2026)
Dots are recorded results; the line is the best result so far; the red dot is the current frontier. Sources for every point are in the table below.
Embed this card

The image names the institutions behind the numbers and links back to this page.

HTML:

<a href="https://canagentswork.com/benchmarks/sop-bench/"><img src="https://canagentswork.com/og/benchmarks-sop-bench.png" width="600" height="315" alt="SOP-Bench: the best result is 72.4% (Claude 4 Opus (ReAct agent), 23 Feb 2026)." loading="lazy"></a>

Markdown:

[![SOP-Bench: the best result is 72.4% (Claude 4 Opus (ReAct agent), 23 Feb 2026).](https://canagentswork.com/og/benchmarks-sop-bench.png)](https://canagentswork.com/benchmarks/sop-bench/)

What it measures

Task success rate (TSR): the share of tasks where the agent's final decision matches the ground truth after it runs the procedure with tools. 2,411 tasks in total (90 to 327 per procedure). The paper also reports execution completion rate (ran without error) and the conditional success rate of completed runs; TSR is their product. Two baseline agents (function calling and ReAct) were run on 11 models; scores are the mean TSR over the 13 procedures.

Task success rate: Share of tasks with the correct final decision, averaged over the 13 procedures. Higher is better.

Status compares the frontier with a ceiling of 100%.

Facts

Grain
Task level: bounded tasks with a clear spec
Environment
api-tools
Grading
automated-tests
Tasks
2,411
Contamination
The tasks, tools, and answers are on GitHub and Hugging Face, and the paper names no held-out set. Data and tools were generated with Claude 3.5 Sonnet v2 and validated by experts.
Reuse
CC BY-NC 4.0 on the repository LICENSE and the Hugging Face dataset card. No commercial use. (open-other)

Limits to keep in mind

  • No leaderboard. The README says detailed results are "coming soon" and gives approximate averages (function calling about 64%, ReAct about 55%) that do not match the paper's tables. The paper's own text gives 73.8% for the best ReAct run where its table and abstract give 72.4%. Source
  • Results cover only two simple baseline agents; the authors say they are not optimal and that the goal is not to rank models. Newer models scored lower than older ones on the ReAct agent. Source
  • Many tasks end in a classification or routing decision (for example a packing group or an email intent) checked against one ground-truth value, so a pass measures the decision, not a work product. Source
  • The best model evaluated is Claude Opus 4.5; no 2026 frontier models have published scores. Source

Recorded results

A selection that shows the frontier over time. The full leaderboard is at the source.

SystemTask success rateDateSource
Claude 4 Opus (FC agent)68.2%23 Feb 2026Amazon · primary
Claude 4.5 Sonnet (ReAct agent)63.3%23 Feb 2026Amazon · primary
Claude 4 Opus (ReAct agent) · frontier72.4%23 Feb 2026Amazon · primary

Where sources disagree

The SOP-Bench paper gives two values for its best result, Claude 4 Opus with the ReAct agent. The abstract and Table 6 say 72.4% task success rate; the text of section 5.1.2 and the conclusion of the KDD camera-ready say 73.8%. The README's approximate "ReAct agents: ~55% average" is an average over all models, not the best.

  • 72.4 — arxiv.org (primary) · Abstract and Table 6 (0.724), arXiv v2 and the KDD 2026 camera-ready PDF.
  • 73.8 — www.amazon.science (primary) · Section 5.1.2 text, "ReAct performs best with Claude 4 Opus as well (73.8% TSR)", and the conclusion, "Claude 4 family: 73.8% vs. Claude 4.5 family: 63.3% average TSR on ReAct", same PDF.

We show 72.4. Status: open.

Timeline

  • 21 Aug 2026 — Amazon releases the full SOP-Bench on GitHub and Hugging Face (CC BY-NC 4.0). Source
  • 9 Jun 2025 — Amazon posts SOP-Bench: agents follow expert-written business procedures with tools. Source

Where it sits in the atlas

Work ladder: Direct evidence for Management and business operations. On the work ladder it counts as 72.4% of tasks completed. How the ladder works

Work it measures (O*NET work activities): Monitor operations to ensure compliance with regulations or standards; Examine materials or documentation for accuracy or compliance; Respond to customer problems or inquiries.

Management and business operationsCustomer support (partial)Office and administrative support (partial)Healthcare (partial)

Procedure execution

Last checked 24 Sep 2026 against 8 primary sources, with a second independent check. See an error? Tell us.

How to cite

Credit the original work first: SOP-Bench by Amazon (https://arxiv.org/abs/2506.08119).

Then, if you used this page:

Can Agents Work. "SOP-Bench: frontier results and sources." https://canagentswork.com/benchmarks/sop-bench/ (accessed 2026-09-24). CC BY 4.0.
@misc{caw-sop-bench,
  title        = {{SOP-Bench: frontier results and sources}},
  author       = {{Can Agents Work}},
  year         = {2026},
  howpublished = {\url{https://canagentswork.com/benchmarks/sop-bench/}},
  note         = {Accessed 2026-09-24. CC BY 4.0}
}