I Gave Five Models the Same Bug and Noted Each Mistake

The bug was a real one we had already solved the hard way: a service that returned stale prices for exactly one customer segment, caused by a cache key that omitted the segment, the multi tenant key defect in the cache that returned another customer's data. After the fix, I froze the evidence as it stood at the start of the investigation, the symptom, the logs, the relevant code excerpts, and the wrong hypotheses we had chased, and handed that identical packet to five current models, asking each for the most likely root cause and the next step. None named the cause outright. All five were useful anyway, because each failed in a way that teaches something about using models as debuggers.

This is the notebook, and the patterns, because the failure modes clustered far more than the successes did.

The setup, and why it was fair

Each model received the same packet: the symptom stated precisely, three log excerpts, four code excerpts including the cache key construction, and the two dead hypotheses we had already eliminated, so the models were not asked to rediscover what the logs disproved. I asked for a ranked list of causes and one next action, and I ran each model five times to see the variance, the discipline from temperature zero is not deterministic. The ground truth was known, which is what made the failures legible.

Failure mode one: the confident wrong first rank

Three of the five put a plausible but disproven hypothesis at rank one, most often "the database returned stale data", and then spent their reasoning defending it. The packet included a log line showing the database read was fresh, and the models cited other lines while not engaging that one. The pattern is confirmation directed at the first generated candidate: the model commits, and the rest of the output is advocacy. The practical lesson is structural, not corrective: ask for the evidence that would disprove each ranked cause, because the disconfirming question forces engagement with the lines the advocacy skips, which is the falsification habit from every bug is a wrong assumption, delegated.

Failure mode two: the missing line

Two models produced excellent rankings that omitted the actual cause entirely, because the cause lived in the one code excerpt they weighted least, the two line key construction, which reads as boring. The models attend to the dramatic, the stack traces and the errors, and underweight the quiet line, which is the opposite of the human expert heuristic that the bug is usually in the boring line. The lesson is about the packet, not the model: when handing code to a model, annotate nothing, but include the quiet lines with the same prominence, and ask explicitly which line, if wrong by one token, produces exactly this symptom, because the question redirects attention to the quiet line.

Failure mode three: the unrunnable next step

Four of the five suggested a next action that was a paragraph of prose, restart with logging, add tracing, which is the advice shaped for a human postmortem, not a step that reduces the hypothesis space now. The one model that suggested a concrete discriminator, query the cache directly for the two segments and compare, was the one model whose rank two was the truth. The correlation held across runs: models that proposed a discriminating observation reasoned better than models that proposed general hygiene. The lesson is to demand the discriminator, the single observation that splits the top two causes, because asking for it changes the quality of the reasoning, not just the format.

Failure mode four: variance at the top

Running each model five times, two of the five changed their rank one between identical runs, which is the boundary case variance from the determinism post, and it means a single run of those models is a sample, not an answer. The stable models were not the larger ones, they were the ones whose top cause was anchored by a specific cited line, which suggests citation is the behavioural signature of reliability, and a model output that cites the exact line it is reasoning from deserves more trust than one that reasons in the abstract, a heuristic I now use as a triage rule.

What the models were good at, reliably

The consistent wins were breadth and speed in the enumeration phase. All five surfaced at least one cause my own first list had missed, usually an infrastructure angle, a proxy cache, a replica lag, that was wrong here but worth thirty seconds to exclude, and all five were fast enough that running five of them cost less than one coffee. The models are a hypothesis generator with a recall bias toward the plausible, and used that way, as the widening step before the human narrows, they earned their place in every run.

The workflow the notebook produced

The packet includes the dead hypotheses, because models waste less time re proposing them. The prompt asks for ranked causes, the disconfirming evidence for each, the quiet line most likely wrong by one token, and one discriminating observation, because those four questions counter the four failure modes directly. Outputs are run more than once when the top rank matters, and trust tracks citation. And the human still owns the narrowing, because the narrowing needs the context the packet cannot carry, which systems this lives in, which incident shaped it, the ownership from AI wrote it so nobody owns it, applied to debugging.

The rule

Models debug the way they write: fluently, plausibly, and without the context that makes a cause true. Use them to widen, demand disconfirmation and discriminators to narrow, trust citation over fluency, sample more than once when it matters, and keep the quiet lines prominent in everything you hand them.

The five models did not find my bug. They made me find it faster, by being wrong in five instructive ways, and a tool that is wrong in instructive ways, at the speed of a keystroke, is a good colleague, as long as the notebook stays in your hand.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud