The score improved. Data was still missing. What a green test suite won't tell you about integration.

A coding agent can already propose the adapter change. That is the part that got commoditized. What the surrounding workflow still has to do is connect the change to an observed problem, define what acceptance means, check that the sources already working still work, and keep the evidence a reviewer will need. Generating the diff leaves all of that undone, and it is where the engineering actually lives.

Ratchet exists so a team can take an observed integration failure through an agent-assisted change with explicit scope, independent evaluation, and a reviewable record of the result. This post follows one of those changes end to end: onboarding a new partner source into a canonical schema.

A partner sends 1,931 company records with anonymized column names. An agent maps three of them into the canonical schema. Schema-mapping F1 rises from 0.50 to 0.91. The test suite is green. The regression check doesn't fire. I approve it at the review gate. Then I look at the data: the founding-year column produced a value for zero of the 1,931 records.

The run above is what happens when that last word, result, is defined slightly wrong. None of it is exotic: the failure survived a green test suite, an independent evaluator, and a human reading both. That is a careful process, not a careless one, which is exactly what makes it worth writing down.

Why the score was right and the integration was not

It would be satisfying to call this a metric that lied. It isn't, and the real explanation is more useful. schema_f1 scores the declared correspondence between a source column and a target attribute. That correspondence was correct. Attribute_6 really is the founding date.

Whether any record survives normalization is a different question, and schema_f1 never asks it. The source ships dates as "1908-01-01"; the chosen normalizer, to_int_year, parses a bare year. Every row raised a ValueError. The change was scored on the half that was right.

The agent wrote down the failure, and it shipped anyway

Here is the part that changed how I think about review gates. The implementer documented the problem itself. It wrote a test called test_fullcontact_attribute_6_full_iso_date_is_invalid_value_format. That is a passing test whose entire assertion is that its own mapping produces INVALID_VALUE_FORMAT on real input, with a comment noting that the normalizers were out of scope for that cycle.

Three controls worked exactly as designed and the combination still landed a change that improved nothing: an oracle scoring a real property that wasn't the property that mattered, a test suite honestly describing a non-functional result, and a human reading both. That isn't an argument that gates are worthless. It's the measured limit of what a gate can do when the number in front of it is answering an easier question than the one being asked.

The record is two commits: f0038e0 mapped the fields, and 0067b7a added the requirement written in response.

A metric that needs no answer key

The run exposed the gap. I added per-field yield: for each mapped target attribute, the share of records that actually produced a value. It was nearly free to build, because evaluate_source already ran the adapter over every record and was throwing the per-field normalization failures away.

What the score couldn't see. Run against the adapter that shipped in f0038e0, and again after the fix.

Pointed at the shipped adapter, it found the known failure at 0 of 1,931, and something nobody knew. country was producing a value in 1,200 of 1,931 records, 62%. At schema_f1 0.91 that mapping had been reported as simply correct.

The property that makes yield the right primary signal is that it needs no gold labels. You don't have to know the correct mapping to know that a field you just declared produced nothing. So it works on day one of an engagement, before any answer key exists. On the real splits it's the only regression signal that functions at all, because fully_correct_rate is None there and the pre-existing check could never fire.

Its limit is the exact mirror of the one it caught, and it belongs in the same breath: yield says a field produced a value, not that the value belongs there. A column mapped to the wrong target with a working normalizer yields 1.00. The two metrics answer different halves of the question and neither replaces the other. That's the lesson, rather than "I fixed it."

What the loop holds, and what it cannot

A ratchet advances and holds. That is what the loop does: bounded changes, a regression check, two human gates, and no cycle that silently gives back ground won earlier. It is also the honest limit, because a ratchet holds whatever position it is handed. The loop can only protect the properties it measures. It held 0.91 for a field that produced nothing, faithfully, for as long as 0.91 was the only thing being watched.

Which is why the fix wasn't a better model. A submission now has to carry a per-field trace of a real source value through its chosen normalizer, and a row whose result is an error is rejected before the tests run. A mapping that normalizes nothing can't be submitted without the agent writing down that it's dead. Re-run from an empty adapter, the same three fields reach the same schema_f1 of 0.91, with founded at 1,056 of 1,931 and country at 1,421. The remaining 875 founding values are empty at the source, which is why 100% was never the target.

Keeping the proposer away from the evidence

The implementer cannot write. Its agent definition grants it Read, Grep and Glob, and it returns {file, old_string, new_string} edits that a single script applies. Every target path is validated and every old_string must occur exactly once before anything is written; one bad edit rejects the whole submission. That replaced a unified-diff handoff, where a read-only agent with no shell was being asked to hand-compute hunk headers it had no way to verify. The contract changed instead of the retry count.

The rule underneath all of this is narrower than "the agent can't grade itself": the component proposing an answer does not control the evidence used to accept it. The implementer reads real records and proposes mappings; it never touches the gold, the fixtures, or the receipts a reviewer reads at the gate.

That rule is the only thing the three systems on this site have in common. Assay applies it to a benchmark verdict, and Vendor vs Valor to a cited research claim. I did not set out to build the same idea three times; each one arrived at it from a different failure.

Read protection and write protection are separate lists, because the implementer has to read the engine it edits against and must never read the answer key. The read boundary used to be a repository-wide deny rule, and that was a bug: it also blocked the orchestrator from the receipts it has to show a human at the gate. It's now a PreToolUse hook declared in the implementer's own agent definition, so it binds that subagent and nothing else. It walks the directory rather than matching its name, because the interesting attack is a grep aimed at a parent whose name matches no pattern. If no configuration resolves, it denies everything. Not configured must never read as not protected.

What this does not establish

• One source, one operator, one machine. The delivery numbers are a cost floor for a reviewed cycle, not a benchmark.
• The protection model is local and tool-level. It is not an operating-system security boundary.
• Yield is a necessary signal, not a sufficient one. It cannot tell you a value landed in the right column.
• The loop needs a human for scope and acceptance. It does not merge or deploy on its own.

The failure above is the strongest thing in the repository, so it stays on the record with its commits rather than being quietly re-measured away. A loop that only publishes its wins is measuring the same easier question all over again.

The full write-up, the receipts, and the evaluator live in the case study. The repository is licensed GNU AGPL-3.0.