How to Audit a Coding-Agent Benchmark Claim

Scope · 7 topics

• Reconstructing a vendor claim and marking missing methodology
• Distinguishing benchmark, variant, harness, scaffold and model
• Recording model versions, evaluation dates and execution budgets
• Separating attempts per task, turns, batch selection and aggregation
• Exposing a hypothetical full-set versus filtered-subset comparison
• Assessing repository relevance without forecasting productivity
• Documentation limitations and monthly canonical maintenance

Direct answer: audit the setup, then assess repository fit

Do not treat a headline percentage as a forecast for your repository. First reconstruct the specific claim: benchmark and version, variant or subset, evaluation harness, agent scaffold, model version, evaluation date, attempts allowed per task, and how the reported result was selected or aggregated. Mark each field as stated, missing or ambiguous. Then separately assess whether the tasks and working conditions resemble your own.

This is a documentation-based walkthrough. Account-level steps were not exercised by CitedIndex, and CitedIndex did not run any benchmark. Source-check date: 9 September 2026.

The checklist below is CitedIndex's editorial audit method, not a claim that either benchmark mandates every disclosure. Where the supplied benchmark documentation does not establish a rule, leave it unknown rather than importing a rule from another benchmark or an agent SDK.

1. Identify what the percentage measures

SWE-bench defines “% Resolved” as the percentage of task instances solved. Source: SWE-bench. Start by copying the vendor's exact metric label, score and linked evidence into your audit record; do not silently rename another benchmark's metric “resolved rate.”

SWE-bench describes its original benchmark as 2,294 instances drawn from real GitHub issues in 12 Python repositories. Source: SWE-bench. SWE-bench describes Verified as a human-filtered subset containing 500 instances. Source: SWE-bench. SWE-bench describes Lite as a 300-instance subset curated for less costly evaluation. Source: SWE-bench.

Editorial audit rule: write down the exact variant and denominator before comparing percentages. Ask for the evaluated task IDs and exclusions, not just the benchmark family name. Treat a vendor-selected subset as a separate population until its relationship to the named benchmark is documented; do not assume it is an official variant.

2. Separate benchmark, harness, scaffold and model

Use these working definitions for your audit. They distinguish responsibilities; they are not presented as terminology mandated by the supplied homepages.

Layer	Working definition	Evidence to request
Benchmark	The task collection and definition of success.	Name, release, variant, task IDs, denominator and exclusions.
Harness	The machinery that prepares evaluation environments, executes submissions and records judgments.	Repository or package, revision, environment configuration, grading configuration and treatment of execution failures.
Scaffold	The agent program around the model: instructions, tools, context handling, control loop and recovery behavior.	Agent name and revision, prompts, available tools, context policy, stop conditions and budgets.
Model	The underlying model invoked by the scaffold.	Exact model identifier or snapshot and inference settings, rather than only a product name.

SWE-bench describes its Bash Only view as the default Verified view, with every model in the same mini-SWE-agent environment. Source: SWE-bench. Editorial implication: preserve that environment label when recording a result; do not collapse it into an unspecified Verified result.

The Terminal-Bench page labels model and agent separately. Source: Terminal-Bench. Use that distinction when transcribing a claim, but do not treat an agent name alone as a complete scaffold specification.

Why a scaffold change matters even with the same model

The OpenAI Agents SDK documents a hook that can edit prepared model input immediately before a model call. Source: Running agents. It also documents a recovery mode in which an unresolved function-tool call produces an error output and the runner invokes the model again. Source: Running agents.

Interpretation: treat a score as a property of the benchmark, harness, scaffold and model together, under the declared budgets and aggregation method. A scaffold-only change can plausibly move the score without changing the model, because it can change what the model receives and whether execution continues. This is a methodological inference, not a measured improvement or evidence that any listed benchmark entry used this SDK. Do not attribute a score difference to the model alone unless the comparison controls the other layers.

3. Build a claim-to-evidence record

For each row below, save the vendor's wording and an evidence location. Use “missing” when there is no disclosure and “ambiguous” when a label permits more than one reading. These are editorial evidence requests, not undocumented benchmark submission requirements.

Field	Question to answer	Do not substitute
Benchmark and subset	Which release and exact task population produced this number?	A family name or an unexplained “filtered” label.
Harness	Which revision, environment and grading configuration were used?	The benchmark name alone.
Scaffold	Which agent revision, instructions, tools and context policy were used?	A model identifier.
Model	Which snapshot and inference settings were used?	A vendor or product family.
Date	When was the evaluation run, and when was the claim published?	Assuming the publication date is the run date.
Attempts per task	How many independent starts were permitted? What counted as a retry, and was state reset?	A turn limit or number of tool calls.
Selection and aggregation	Is this one evaluation batch, a mean across batches, the best batch, or success on any of several attempts per task?	The word “score” without an aggregation definition.
Denominator and failures	How many tasks were included, and how were timeouts, invalid outputs and infrastructure failures counted?	A percentage without counts or exclusions.
Execution budget	What limits applied to time, tokens, turns and tools?	An assumption that both results had equal opportunity.

The OpenAI Agents SDK describes its max_turns limit in terms of agent-loop turns, meaning LLM calls. Source: Running agents. Editorial rule: record turns within an attempt separately from fresh attempts at the task. The SDK's limit is not a benchmark attempt policy.

Evidence boundary: the supplied SWE-bench and Terminal-Bench homepage extracts do not establish an attempt-count, retry, aggregation or failure-handling rule for the claim you are auditing. They are silent on those rules at the level needed here. Request the specific evaluation methodology; do not assume one attempt, infer the meaning of an uncertainty figure, or borrow a policy from the SDK documentation.

4. Worked example: a large improvement with two changed conditions

Hypothetical example only: an invented vendor advertises, “Coding-agent score jumps from 40% to 65%.” All benchmark names, identifiers, dates and results in this table are invented; no behavior is attributed to a real vendor.

Audit field	Earlier result	Advertised result
Benchmark	ExampleCode 1.0	ExampleCode 1.0
Task population	Full set: 1,000 tasks	Vendor-filtered subset: 200 tasks
Reported successes	400 of 1,000: 40%	130 of 200: 65%
Harness	ExampleHarness revision A	ExampleHarness revision A
Scaffold	ExampleAgent revision B	ExampleAgent revision B
Model	ExampleModel snapshot C	ExampleModel snapshot C
Evaluation date	1 August 2026	1 September 2026
Attempts allowed per task	One fresh attempt	Up to five fresh attempts
Per-task success rule	The single attempt succeeds	Any of the five attempts succeeds
Batch selection	One batch, no selection across batches	One batch, no selection across batches
Still missing	Time and token budgets	Filtering rationale; time and token budgets

Interpretation of this hypothetical: the arithmetic shows a 25-percentage-point difference, but it does not isolate an improvement in agent capability. Both the task population and the opportunity to succeed changed. “One batch” does not make the second figure a one-attempt result: its success rule selects success across as many as five attempts per task.

Decision: label the comparison “not like-for-like,” not “false.” Ask for both configurations evaluated on the same task IDs with the same attempt allowance, budgets and aggregation rule. Do not try to normalize 65% back to a one-attempt full-set result using the headline figures alone.

5. Decide what remains relevant to your repository

Editorial interpretation: a resolved rate on a defined public task set is evidence about that evaluation, not a measured success rate on your private code. A curated subset does not establish coverage of your unstated conventions, internal dependencies or private acceptance suite. Do not convert its percentage into an expected productivity gain.

Before deciding whether to conduct a local evaluation, make a separate repository-fit record:

• List representative work you actually need: for example, issue fixes, cross-package changes or migrations. Mark these as your requirements, not benchmark capabilities.
• Compare your languages, dependency setup and task descriptions with the documented benchmark population. Record uncovered requirements as unknown rather than assumed matches.
• Identify conventions the agent would need but that the task text does not state. Decide what instructions and context you would provide in a local evaluation.
• Identify acceptance checks from your private suite that are not represented in the benchmark evidence. Keep functional acceptance, review effort, elapsed time and operating cost as separate local questions.
• Check whether the evaluated scaffold and budgets are available in the product configuration you would use. If that equivalence is undocumented, record it as unknown.

End with one of three editorial dispositions: “insufficiently specified” when essential fields are missing; “not like-for-like” when a comparison changes material conditions; or “sufficiently specified to inform a local evaluation” when the setup is documented and the task fit is plausible. None is a product recommendation or a promise of usefulness.

Limitations and maintenance

This guide audits disclosure and comparability; it does not independently establish that a reported run happened, that grading was correct, or that a result transfers to your repository. No current leaderboard position is reproduced. The SDK documentation supplies examples of runtime controls, not SWE-bench or Terminal-Bench evaluation rules.

Review this canonical guide every 30 days. If a benchmark definition or relevant methodology changes, correct the affected explanation and its evidence here rather than creating a new dated page. A new source-check date alone is not a substantive revision.

Sources

• SWE-bench: metric, task populations and evaluation-view descriptions
• Terminal-Bench: separate model and agent labels
• OpenAI Agents SDK: Running agents, runtime controls and recovery behavior