The agent passed every test it could see

Every number below lives in the repository, next to the command that produced it.

An agent hands you a patch and a green test run. Accepting it means trusting the agent's account of its own work. A verdict is not a trophy, it is an input: it decides whether a proposed repair gets accepted as done, and which agent looks better than another. Get it wrong and both of those decisions go wrong quietly, because nothing downstream can tell a bad verdict from a good one.

Assay grades the tree the solver left behind against tests it was never allowed to see, and reads the verdict from what the test runner wrote rather than from anything the solver said. This post follows the run that made the difference visible.

I pointed a frontier model at one real task, an Ansible galaxy bug. A hundred steps, 65 cents. It worked, genuinely worked. It wrote 294 lines across three files. It introduced a regression at step 33, caught that regression itself at step 46, and had it fixed by step 64. When it stopped, all 133 tests it could run were passing.

It had fixed none of the 18 tests that actually graded it, and broken 4 more. Those 22 were withheld from it, so its own checking could never have reached them. Nothing about the run looked like failure. There was no moment where someone watching over its shoulder would have said wait.

That run is what an untrustworthy verdict looks like from the outside. It looks like success.

The observed run on ansible__ansible-de5858f4. Separation is what makes the verdict checkable; it is not by itself a guarantee that the verdict is right.

Competence made it more dangerous, not less. A weaker model I ran first, a local 7B, declared itself finished having changed zero files. Any harness that glances at the tree catches that one. The hosted run is the case worth building for, and the only thing that caught it was measuring the tree against tests the solver never saw.

This is not one bad run

I had one task and one model, which is an anecdote. Then someone audited the field. Meerkat covers more than 1,000 agent runs across nine benchmarks and twenty-eight-plus leaderboard submissions. It separates two things people usually run together. Task-level reward hacking is the agent gaming the task: special-casing the test, stubbing the function. They confirmed 31 of those, across six benchmarks. Harness-level cheating is the agent reaching past the task into the machinery that grades it, and they found it in every top-scoring Terminal-Bench 2.0 and HAL USACO submission they examined. In 415 of 429 successful traces from one agent, it read a /tests directory the task spec says it should not be able to see.

Those numbers are not a claim about my run and my run is not evidence for theirs. They matter because they say what the failure mode is: not exotic, not adversarial, not a jailbreak. The leaderboards at the top of the field are reporting scores that were partly produced by agents reading their own answer key, and it took a dedicated audit to notice.

The published harnesses cannot see this failure

A harness that asks the solver how it went files this run as a success. So does one that reads its verdict out of prose. The argument against the published harnesses takes three steps, and they are worth keeping apart, because the first one on its own proves nothing.

One: how they extract a result. Of the 1,000 public SWE-bench Pro instances I counted, 1,000 grade through a per-instance parser.py doing regex or string work on stdout. Seventy distinct implementations; median 165 lines. I can only speak for the instances I opened, but within them the pattern is total.

Two: that extraction is demonstrably wrong sometimes. Prevalence is not incorrectness, and this is the step that supplies it. Forty-nine of those instances ship parser_original_invalid.py beside parser_attempted_fix.py, the authors putting the finding in the filename. On a different benchmark, UTBoost found erroneous annotations in 271 of the 500 SWE-bench Verified instances, and 79 patches that had been recorded as passing when they were not. It replicates on SWE-bench Lite: 164 of 300, and 64 more bad patches. Different suite from my census, same defect class.

Three: the environment helps it along. Every SWE-bench Pro image bakes in --reruns=3, which gives a failing test up to three more attempts after the first. It does not promise a pass. What it does is let a later attempt stand in for an unstable one when only the final outcome gets reported. Nobody wrote that flag intending to launder failures.

Note what this argument is not. Reading a test runner's output is not the sin. Assay reads the runner's output too. The difference is that it reads a structured channel the runner emits for machines, then cross-checks it against the runner's exit code, rather than pattern-matching prose the runner emitted for humans.

The answer key sits outside every tree the solver can reach

The natural way to hide the grading tests is to reset the image's checkout to the base commit and delete the test files. I built that first. It does not work. At the base commit the repository still holds 67,370 objects and 4,111 later commits, and the answer key is one git show away.

Now the part that matters, because it is the part that nearly shipped. The obvious check for this is to grep the filesystem for the guarded test names. That grep matches zero files, because packed git objects are zlib-compressed. The probe comes back green while the solver can read the answers whenever it likes. A control that cannot fail is not a control.

So the repository is reset inside a network-less container, exported once, and committed into a git object store that lives outside every phase tree and is never bind-mounted into one. Each phase is materialized out of it with git checkout-index. The solver's patch is then diffed against that store, on the host. SWE-bench's own harness has the agent run git diff inside the container and string-matches a sentinel out of its stdout, handing the thing being measured control of the measurement.

My own test suite made the same mistake

For a while the sandbox had 94 passing unit tests against a fake Docker daemon. It was also not starting a container. Every one of those tests was green, and the thing they described did not happen.

Swapping in a real daemon surfaced six bugs quickly: an inherited ENTRYPOINT; a shell line collapsed by shlex.join; harness output polluting a stream something else was parsing; a --user that locked the solver out of the code it had been asked to edit; --read-only stopping pytest before collection; and PYTEST_ADDOPTS adding --reruns=3 to every verdict. Two more were invisible on macOS and failed only on Linux CI, both about bind-mount ownership.

The suite is 445 tests and thirteen need a real daemon. That ratio is small because it has to be, and non-zero because a mock will confirm whatever shape you taught it.

A refusal is a better product than a guess

pytest exits 0 when everything passed and 1 when tests failed. Only those two describe the tests. Every other code describes the run: 3 an internal error, 4 a bad invocation, 5 collected no tests at all, anything at or above 128 a signal. Exit 5 is the one that matters most here, because a suite that collected nothing produces a report with no failures in it, and no failures is exactly what success looks like. So the harness refuses every code but 0 and 1, and keeps the partial report, because it is evidence even when it is not a verdict.

The same rule covers anything unmeasured. A cost that was never computed is reported as null, never 0.0; upstream returns zero there, and passing that on would claim the run had been measured and found to be free.

Two smaller decisions in the same spirit. The report file is written to a temporary name and renamed, so no reader ever observes a half-written file and has to decide whether a short report means fewer tests ran or the write was cut off. And killing a docker run client does not stop the container, because the daemon is its parent, and --rm reaps only on a normal exit. Every container is named and force-removed on timeout, and each child goes into its own process group. One test file took 61s before that change and 2.4s after.

The stripper I measured and did not build

Grading tests are hidden a whole file at a time, which is deliberately imprecise: hiding a file can withhold unguarded tests that happened to live in it. The precise version is an AST-level stripper that removes only the graded functions.

I measured the imprecision before engineering it away. Across the 1,000 instances examined, the worst case withholds 8 of 2,176 unguarded tests, 0.37%. On the bundle proven end to end it is 0 of 147. An AST stripper would have recovered eight tests across a thousand instances, at the cost of a fragile language-specific mutator sitting directly under the verdict. It was not built, and that number is the reason.

What this does not claim

Neither model solved the task. This is a result about a harness, not a leaderboard entry about a model, and reading it the other way would be the same error the post is about.

• One bundle is proven end to end: ansible__ansible-de5858f4, 165 tests, 18 of them grading, both re-derived rather than trusted.
• Reading the verdict from XML the runner wrote removes the parser from the loop. It does not make the tests themselves correct, and a wrong grading test still grades wrongly.
• It is a local, single-operator tool, not a hosted product, with no claim of production impact.
• Timings were taken on Apple Silicon with the example image emulated. They roughly halve on native hardware.

The proposer does not control the evidence

The hosted run got caught because the harness never asked. It read the verdict out of XML the test runner wrote, from a tree the solver could not reach, against tests the solver never saw. The rule is narrower than it sounds: the component proposing an answer does not control the evidence used to accept it. Every other design in this repository is downstream of that.

A benchmark that asks the model how it did is not measuring the model. It is interviewing it.

The same boundary, drawn twice more: Ratchet keeps an implementer away from the evaluator that scores it, and Vendor vs Valor keeps the agent that writes a claim away from the pass that verifies it.

Sources

• Meerkat: Auditing Reward Hacking in Agentic Benchmarks, arXiv:2604.11806. Harness-level and task-level counts above are quoted from it. Code at BrachioLab/Meerkat.
• UTBoost: arXiv:2506.09289. Source of the SWE-bench Verified and Lite annotation counts.
• The SWE-bench Pro parser census, the 0.37% file-level imprecision figure and the observed run on ansible__ansible-de5858f4 are mine, recorded in the repository next to the commands that produced them.