When an Agent Reviews an Agent, the Praise Is Guaranteed

Two models from the same family reviewed each other's code and agreed it was fine, because they share the same priors about what plausible code looks like.

Share
When an Agent Reviews an Agent, the Praise Is Guaranteed. Abstract code review illustration in orange and dark grey on debugly.dev

The pipeline had two agents, one writing and one reviewing, and the review comments were courteous, specific and useless. The reviewer praised the structure, suggested a rename, and approved, and the approved code contained the defect the whole exercise existed to catch, a swallowed error path, which the reviewer's own training distribution also considers a low priority smell. The pair was not a review. It was a mirror, and the mirror nodded.

This is the echo chamber problem in agent on agent review, and the reason it is dangerous is that it produces the artefacts of rigour, comments, approvals, a green gate, without the property rigour exists for, an independent judgement that can disagree.

This was an author agent and a reviewer agent from the same model family, and the failure generalises to any pair whose omissions are correlated.

Why agreement is not evidence

A review is informative only to the extent that the reviewer can see what the author missed, which requires the two to be differently wrong. Human review works because humans differ in experience, attention and prejudice, and the diff between their perceptions is where the catch happens. Two models from the same family, trained on the same corpus, share their omissions. The error path the author under weights is the error path the reviewer under weights, because the weighting is the same learned prior.

So the probability that the reviewer flags the author's blind spot is only marginally better than the probability the author flagged it themselves, and the approval carries almost no information. The gate is green, and the green is noise. This is the statistical core of the single agent self review defect in an agent should never merge its own pull request, extended to two bodies with one mind.

The shape of the useless comments

The echo chamber has a recognisable output style, and naming it helps spot it in a review log.

The comments are about surface: naming, structure, idiomatic patterns, because surface is where the shared prior is strongest and easiest to verbalise. The comments avoid the semantic edges: the failure path, the concurrency, the rollback, because those are where judgement is needed and the prior is thinnest. And the verdict trends positive, because a model reviewing plausible code finds it plausible, and plausibility is the prior, not an analysis.

The result is a review that reads like a style guide passed over the diff, which is the same failure as a test that restates the implementation is not a test, translated to prose: the reviewer is checking the code against the average of code, not against the system's requirements.

When two agents can add signal

I do not want to claim agent review is worthless, because under the right design it adds real signal, and the design is about breaking the correlation.

Different distributions. A reviewer from a different model family has different priors and therefore different omissions, and the overlap of what the two miss is smaller than either alone. Independence is the property to buy, and it is bought with heterogeneity, not with a second instance of the same model.

An adversarial brief. A reviewer asked "what does this break and what must it never do" behaves differently from one asked "review this code", because the adversarial frame overrides the plausibility prior and directs attention to the edges. The brief is the cheapest decorrelation device available.

Grounding in execution. A reviewer that runs the diff, or reads the failing test the author claims to fix, reasons from evidence rather than from the prior, and evidence is uncorrelated with the author's omissions. The differential check, does a test fail without the change, is the strongest review signal any agent can produce, because it is a fact.

A checklist of the known failure classes. Giving the reviewer the team's list, swallowed errors, unbounded retries, missing rollback, unvalidated boundaries, turns the review from open ended judgement into targeted inspection, and targeted inspection catches what the prior would skip.

The human stays as the arbiter of the semantic

Even a decorrelated agent pair is two machines sampling plausibility, and the semantic question, does this change do what the product needs, lives outside both. The human's role in an agent heavy pipeline is not to re read every diff, which does not scale, but to arbitrate the disagreements between the agents and to own the semantic sign off, because the disagreement between two different models is precisely the signal that a human should look, and agreement between them is the signal that the change is, at least, unsurprising.

That inversion, review the agreements lightly and the disagreements heavily, is the allocation of scarce human attention that agent pipelines need, and it is the opposite of the current habit, which is to trust the green gate.

If it happens again

Agreement between correlated minds is not evidence, and two agents from the same family reviewing each other produce the artefacts of rigour without the judgement. Break the correlation with different models, adversarial briefs, execution grounding and targeted checklists, and point the human at the disagreements, because the disagreement is the only honest signal in the loop.

The mirror that nods is not a reviewer. A reviewer is a mind that can be wrong in a different way, and the difference is the whole value, in code review as in blameless postmortems, same mechanism, different costume.

It is worth naming the metric that exposes the chamber, because a gate needs a number. Log, for every agent review, whether the reviewer produced at least one semantic objection or a request for evidence, and chart the rate over time. A reviewer that never objects is not rigorous, it is decorative, and the objection rate is the cheapest proxy for whether the second mind is actually a second mind. When the rate collapses after a model or prompt change, you have measured the echo arriving, before the incident prices it.


Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud