Agents · Research analysis
A coding agent is a system, not a score
How to compare coding agents without confusing model quality, scaffolding, test contamination, and retry budget.
On this page
THE ESSENTIALS
- Compare a frozen system configuration, not only a model name.
- Retries, environment access, and harness versions belong beside the result.
- Measure regressions and intervention burden as well as task resolution.
Our analysis
The comparison problem
Two agents can report different resolution rates while using different tools, token budgets, retry policies, and environments. The difference may be meaningful, but it cannot automatically be attributed to the underlying language model. The right object of evaluation is a reproducible system configuration.
For an agent intended to complete terminal work, the practical question is whether it can interpret a task, inspect an environment, change the right files, verify the outcome, and stop accurately. A patch that happens to pass a narrow test is only one part of that behavior.
Published evidence
What the benchmarks actually describe
SWE-bench Verified contains 500 human-filtered instances. The official site distinguishes arbitrary agent systems from a shared minimal agent environment and warns that releases of that environment are not necessarily directly comparable. This makes the harness part of the reported result. [1]
Terminal-Bench’s current site presents version 4.0, while its run guide uses the Harbor framework. Benchmark versions should be pinned rather than treated as a timeless dataset name. Its explicit separation of benchmark data from training is also important for credible evaluation. [2, 3]
The SWE-bench repository documents containerized evaluation and result caching behavior. Reusing an identifier can reuse earlier results, so a changed prediction needs a fresh evaluation identity where the harness requires it. That is a reproducibility issue, not a model capability issue. [4]
Proposed experiment
A comparison protocol we would trust
Define two tracks. In the model track, hold the agent loop and tool interface fixed while varying the model. In the product track, allow each system its intended tools and strategies but equalize the operational budget. Publish both; do not mix their conclusions.
Freeze a manifest containing the model identifier, agent revision, task-set version, container digest, dependency lockfiles, decoding settings, maximum wall time, token budget, network policy, and retry definition. Record the initial repository state and preserve the full action log. A rerun should start from a clean environment rather than inherit a previous attempt’s artifacts.
Score the final state using the benchmark’s authoritative evaluator, plus independently specified regression checks. Keep environment failures as a separate category with evidence. An unexplained crash should not silently disappear from the denominator. If the agent declares success without adequate verification, count that behavior separately even when a task happens to pass.
| Metric | Why it matters | Report with it |
|---|---|---|
| Resolved tasks | Task-level effectiveness | Exact evaluator and task version |
| Wall time and token use | Operational cost of a solve | Median, tail and retry budget |
| Regression rate | Changes outside the requested fix | Predeclared checks |
| False success declarations | Reliability of the agent’s own status | Verification evidence |
| Human interventions | Autonomy under the stated policy | What intervention was allowed |
Analytical example
Why a single success rate is not enough
As a hypothetical illustration, 70 successes in 100 independent trials gives an observed rate of 70%. A 95% Wilson interval is approximately 60.4% to 78.1%. This is a calculation, not a result for any existing agent. It shows how much uncertainty a small evaluation can hide.
Benchmark tasks are not necessarily independent or identically distributed. Several can share a repository, dependency stack, or failure pattern. For a serious comparison, retain task identity and use paired analysis on the same tasks. If uncertainty is estimated by resampling, consider clustering by repository or task family rather than assuming every observation is unrelated.
Best-of-k success and single-attempt success answer different questions. Report both only when k, selection rules, and cost are explicit. A system that spends much more compute to obtain a successful final artifact may be useful, but the gain should not be described as a free improvement.
Proposed experiment
Make the failures useful
Tag failures by the earliest material cause: incorrect task interpretation, navigation failure, invalid action, environment problem, incomplete change, missing verification, or harmful side effect. Have a reviewer inspect a sample without seeing the system identity where practical. Disagreements should lead to a revised rubric rather than forced certainty.
For an internal development suite, introduce held-out repositories and task families, not just different issues from familiar repositories. Keep the official benchmark evaluation separate from training and prompt tuning. Synthetic tasks can exercise the same skills without copying protected evaluation instances.
Our analysis
The result worth publishing
Our recommended deliverable is a result bundle: a frozen manifest, per-task outcomes, resource use, action traces, a failure taxonomy, and uncertainty estimates. The headline number should be a compact index into that evidence, not a substitute for it.
Scope & limits
What has and has not been run
We executed the illustrative Wilson calculation only. No agent was benchmarked in this study, and no leaderboard position is claimed. Current benchmark documentation was checked on the review date; a future run should pin the exact release it uses.
Reproduce the analytical examples
The standard-library Python program checks the probability, cache-memory and interval calculations, plus the small policy examples. It does not train a model or run the cited benchmarks.
A CLOSER LOOK
Ask this article.
Ask a question to find passages and their references. Answers are extracted from this article; no live AI model is connected.
READ THE ORIGINAL WORK
Sources & references
Sources reviewed 20 October 2026. Papers describe their own models and experiments; our proposed experiments are identified separately.
- 1SWE-bench Verified ↗
SWE-bench team · Current benchmark documentation
www.swebench.com - 2Terminal-Bench ↗
Terminal-Bench team · Current benchmark site
www.tbench.ai - 3Run Terminal-Bench ↗
Terminal-Bench team · Current run guide
www.tbench.ai - 4SWE-bench repository and evaluation guidance ↗
SWE-bench maintainers · Living repository
github.com
Keep exploring
All stories ↗
TrainingGRPO efficiency starts before the kernel
A practical investigation of sparse rewards, loss normalization, and what a faster training step actually proves.
SafetyPrompt injection is a permission problem too
Why an agent’s ability to read a document should not grant that document authority to authorize an action.
SystemsLong context has a memory budget
A worked KV-cache calculation, and why FlashAttention, GQA, and PagedAttention solve different problems.