What it is
This is the family of benchmarks that test agents on real engineering work rather than on a quiz. SWE-bench Verified is 500 human-vetted GitHub issues from popular repos: the agent has to produce a patch that genuinely fixes the bug and passes the tests. Terminal-Bench is ~100 hand-written terminal tasks (build the code, bring up a server, solve an infra problem).
Where it came from
- 2023 — SWE-bench is released (Princeton). 2024 — OpenAI and Anthropic produce SWE-bench Verified, filtering out broken tasks.
- 2025 — Terminal-Bench (Stanford + Laude Institute) adds "working in a terminal" as its own axis; SWE-bench Pro, SWE-rebench and others appear.
- Leaderboards became the public arena where labs compare themselves on every release.
Why it took off
- Older multiple-choice benchmarks were saturating and said nothing about whether a model can fix real code.
- "Issue closed / tests pass" is an honest, verifiable metric that mirrors an engineer's job.
- The numbers became the industry's marketing language: "X% on SWE-bench Verified" is the first thing shown at a model launch.
How to use it today
- When comparing models or agents, look at SWE-bench Verified and Terminal-Bench, not at abstract "IQ tests".
- Remember the scaffold: the same LLM with different harnesses scores differently — compare the whole setup, not the bare model.
- Test your own task on your own code: a public benchmark is a signpost, not a guarantee for your repo.
What to watch out for
- Contamination: tasks from open repos may have leaked into training — hence the work on inflated metrics (SWE-ABS and similar).
- A high score doesn't mean "it will handle your legacy": a benchmark isn't your context.
- The numbers go stale fast — check the date on the leaderboard.