One Reviewer Approved It. The Other Found the Bug.

More review rounds do not converge on zero defects. We measured it: across 24 review ledgers, the per-round finding counts wander instead of falling. One story went 7 4 2 4 1 5 5 4 3 2 2 2 2 2. Another climbed: 6 2 2 3 3 4 5.

That is not a team getting sloppier. It is what happens when review has no stopping criterion, and it took us 222 rounds to see it.

This post is about two measurements from that period and the rule we built out of them. It will not tell you how to make review faster. Both changes we made cost more review, not less.

Two reviewers, four stories, one overlap

We put every story through two independent reviews: an internal architect reading the code against our project rules, and an external reviewer, Codex, reading the same diff cold.

On four consecutive stories in late July, the two reviews produced six blocking findings between them. They agreed on one.

The five that only one reviewer caught were not near-misses by the other. They were different kinds of defect:

• The external reviewer checked arithmetic and vendor contracts. It caught a wrong threshold, an interface that accepted a request it never implemented, and a test that claimed to kill a mutant it did not kill.
• The internal reviewer read the code alongside our project rules. It caught a warning that asserted a monetary fact the calculation one line above contradicted.

Neither list is a subset of the other. A reviewer who knows your project conventions sees structural lies, like a warning that contradicts the calculation above it. A reviewer who knows nothing about them recomputes your numbers. Ask one to do both and it does the one it is shaped for.

The measurement that actually changed our minds

The overlap is a curiosity. The next measurement is why we kept paying for both.

On a later story we ran 34 rounds. The two reviews agreed twice in 34 rounds. On round 33, the external reviewer returned APPROVED, and the internal reviewer found a genuine blocker that same hour.

Single review does not find fewer defects. It hands you a false "done" and you ship on it.

"Fewer findings" is a cost you can budget for. A wrong approval is not a smaller version of the same thing.

What 167 blocking findings were actually about

Two reviewers cost roughly twice as much review. So we went and looked at what all that review was buying.

We took every blocking finding whose full reasoning was still on disk: 167 findings across seven review ledgers, 48 external rounds, about 11.2 million tokens. Then we classified each one.

The first row is the expensive one. 61% of blocking findings were about a test being insufficiently strict, with no executed mutant in the production code behind them. They were not wrong. A stricter test is almost always available, which is exactly the problem: "this could be stricter" has no floor. A reviewer can generate that finding forever, and ours did.

What 167 blocking findings were about. The rows overlap, so they do not sum to 100.

For the same period: 24 ledgers, 222 rounds, 21 hours 21 minutes of external review time. 18 rounds came back APPROVED, or 8%.

The rule we ended up with

Two rules narrow what counts as a blocking review finding:

A finding blocks only when it names an executed mutant in a production file in the diff. An executed mutant is a deliberate change to shipped code that the current tests fail to catch: you make the change, you run the suite, the suite stays green. Not a hypothetical. Not a test that could be stricter. Everything else becomes a task and leaves the review.

Three rounds, then stop. Whatever remains ships as a ticket.

The cap is not a patience threshold. It comes from the ledger data above: if finding counts do not trend to zero, no amount of waiting ends the review. The gate is the only part with a stopping criterion in it, and the cap bounds whatever the gate has not yet caught.

Both rules went in on 25 August, so every series in this post predates them. What follows is what they were a response to, not a demonstration that they worked.

The part that argues against us

A page in our dashboard took eight rounds: 5 4 3 6 4 4 2 0. Round four found more than round three.

Findings per review round, three stories. Only one reaches zero, and it gets there by running out of defects rather than by meeting a criterion.

Almost every finding from round two onward was about a fix from the round before it. Three shapes, all ours:

• A wrong number replaced by a different wrong number, caught on the third round.
• The named line fixed, the class of defect left alone.
• A guard added pre-emptively, which then grew machinery for four rounds, defending against a defect that was not there.

That series does reach zero, on the eighth round, for a page whose product defects were exhausted by the first. Read it as the cost rather than the counterexample: without a stopping criterion, zero arrives by exhaustion. The gate came two weeks later, after series like the fourteen-round one above showed that exhaustion is not guaranteed to arrive at all.

This is the same compounding shape that makes multi-step agents unreliable: each step's output becomes the next step's input, and errors introduced along the way get audited as though they were the original work.

So the review was largely reviewing itself. Then round seven, after we put "a defect in the page matters more than anything else" into the reviewer's prompt, found a real lie on the page: it was showing the wrong photograph.

Six rounds did not find it. The seventh did, once the prompt demanded a page defect rather than a defect in the repairs. Whether the wrong photograph was there from the start or arrived in one of my own fixes, we never established. The last defect on that page was caught by the internal reviewer, not the external one.

I cannot tell you the loop is always productive. On that page it mostly was not. Six disciplined rounds audited their own repairs, and the defect that mattered surfaced only when someone changed what the reviewer was told to look for.

If you are running AI code review

1. Run two reviewers with different inputs, not two passes of the same one. The overlap on our four stories was one finding in six. Two passes of one reviewer would have reproduced one blind spot twice.
2. Define what blocks before you start. Ours is an executed mutant in a production file in the diff. Without a definition, "could be stricter" fills the space, and that was 61% of our findings.
3. Cap the rounds. Not for patience, but because the counts do not converge on their own.
4. Read the round-over-round series, not the totals. A rising series means the review is auditing its own fixes.
5. Never let one approval close a review. Round 33 of our longest story was an APPROVED sitting next to a real blocker found the same hour.

Pull up your last review thread and count the findings per round. If the series is not falling, you do not have a converging review. You have a loop with no exit condition, and the exit is a rule about what blocks — not more rounds.

FAQ

Why run two reviewers instead of one thorough reviewer?

Because the second reviewer covers a different blind spot, not the same ground twice. Across four consecutive stories our two reviews produced six blocking findings and agreed on exactly one. The external reviewer recomputed arithmetic and checked vendor contracts; the internal one read the code against project rules and caught a warning that contradicted the calculation above it. Two passes of a single reviewer would have reproduced one blind spot twice instead of covering two.

What should count as a blocking review finding?

An executed mutant in a production file in the diff: a deliberate change to shipped code that the current tests fail to catch. You make the change, you run the suite, the suite stays green. Without that definition, "this test could be stricter" fills the space — it accounted for 61% of our 167 blocking findings, and it has no floor, because a stricter test is almost always available.

Why cap review at three rounds rather than waiting for zero findings?

Because the finding counts do not trend to zero on their own. Across 24 review ledgers the per-round series wander rather than fall: one ran fourteen rounds without settling, another dipped to 2 by its second round and was back up to 5 by its seventh. Where a series does reach zero it arrives by exhaustion rather than by any criterion, and exhaustion is not guaranteed to arrive at all. The cap bounds the remainder; whatever is left ships as a ticket.

The measurements here are from building Clipwright(opens in new tab), our video generation API, over July and August 2026. The same totality-check discipline shows up in how we classify vendor work for billing.