IaC-Eval

Built by University of Michigan and Cisco Research · released 2024

A code-generation benchmark for Terraform. Each of 458 human-written scenarios pairs a natural-language request for AWS infrastructure with an intent specification. A model's Terraform program passes only if it plans successfully and meets the specified intent.

Frontier

36.7%

pass@1

GPT-4 with retrieval-augmented generation · harness: RAG over Terraform documentation

2024 · Source: University of Michigan (benchmark maintainers)

Best result across the paper's enhancement strategies (few-shot, chain of thought, multi-turn, RAG). Multi-turn GPT-4 scores 31.12%.

Emerging

The best result is at 20–60% of the ceiling, or at 50–100% of human parity.

Dots are recorded results; the line is the best result so far; the red dot is the current frontier. Sources for every point are in the table below.

Embed this card

The image names the institutions behind the numbers and links back to this page.

HTML:

<a href="https://canagentswork.com/benchmarks/iac-eval/"><img src="https://canagentswork.com/og/benchmarks-iac-eval.png" width="600" height="315" alt="IaC-Eval: the best result is 36.7% (GPT-4 with retrieval-augmented generation, 2024)." loading="lazy"></a>

Markdown:

[![IaC-Eval: the best result is 36.7% (GPT-4 with retrieval-augmented generation, 2024).](https://canagentswork.com/og/benchmarks-iac-eval.png)](https://canagentswork.com/benchmarks/iac-eval/)

What it measures

Whether a model can write a correct Terraform configuration from a plain-English request. Generated programs are checked with Terraform tooling and Open Policy Agent rules that encode the scenario's required resources and settings. The headline is pass@1 accuracy over the 458 scenarios, estimated from 20 samples per problem.

pass@1: Share of scenarios where a single generated Terraform program plans and satisfies the intent specification. Higher is better.

Status compares the frontier with a ceiling of 100%.

Facts

Grain: Task level: bounded tasks with a clear spec
Environment: cli
Grading: automated-tests, state-check
Tasks: 458
Reuse: CC-BY-4.0 (Hugging Face dataset card) (open-cc-by)

Limits to keep in mind

Results in the paper cover models from 2024 (GPT-4, GPT-3.5, Gemini 1.0 Pro, CodeLlama, WizardCoder, Magicoder). The README lists the leaderboard as "TBD" and no newer results are published by the maintainers. Source
AWS and Terraform only. Other clouds and IaC tools are left as future work. Source
The maintainers describe the framework as not production-ready and under active patching. Source

Recorded results

A selection that shows the frontier over time. The full leaderboard is at the source.

System	pass@1	Date	Source
WizardCoder-33B-V1.1	8.93%	2024	University of Michigan · primary
GPT-4 (zero-shot)	19.4%	2024	University of Michigan · primary
GPT-4 with retrieval-augmented generation · frontier harness: RAG over Terraform documentation	36.7%	2024	University of Michigan · primary

Where it sits in the atlas

DevOps, SRE, and IT operations

Infrastructure as code

Go to the source

Paper proceedings.neurips.cc
Code github.com
Dataset huggingface.co

Last checked 23 Sep 2026 against 3 primary sources, with a second independent check. See an error? Tell us.

How to cite

Credit the original work first: IaC-Eval by University of Michigan and Cisco Research (https://proceedings.neurips.cc/paper_files/paper/2024/hash/f26b29298ae8acd94bd7e839688e329b-Abstract-Datasets_and_Benchmarks_Track.html).

Then, if you used this page:

Can Agents Work. "IaC-Eval: frontier results and sources." https://canagentswork.com/benchmarks/iac-eval/ (accessed 2026-09-24). CC BY 4.0.

@misc{caw-iac-eval,
  title        = {{IaC-Eval: frontier results and sources}},
  author       = {{Can Agents Work}},
  year         = {2026},
  howpublished = {\url{https://canagentswork.com/benchmarks/iac-eval/}},
  note         = {Accessed 2026-09-24. CC BY 4.0}
}