Code Coverage Tells You What Ran, Not What Was Checked

The repository badge said 92 percent coverage, and the quarter's most damaging bug shipped through a file that was, by every measure, covered. The test executed the function. It asserted a property that was true whether or not the bug was present. The line ran, the coverage counted it, and the bug sailed through a green suite, which is the entire argument in one incident.

Here is the thesis. Code coverage measures execution, and protection comes from assertion, and the gap between the two is where the number stops being an engineering signal and starts being a marketing one. I say this as someone who uses coverage daily, for the one thing it is good at, and distrusts it for everything else.

What coverage actually measures

Coverage instruments the code and records which lines, branches or paths executed during the test run. That is a fact about the tests' reach, not about their judgement. A test that calls a function and checks only that it returns without throwing has executed every line in the function and verified almost nothing.

The metric's honest use is negative. Low coverage identifies code no test ever touches, and untested code is where surprises live, so a coverage report is an excellent map of the dark areas. It is a flashlight, not a judge.

The metric's dishonest use is positive. A high number is treated as evidence of safety, but execution without assertion is not safety. The number cannot distinguish a suite that checks behaviour from a suite that merely visits code, and visiting is the cheaper thing to produce.

Why the number rises while protection stays flat

The incentives do it. Coverage is cheap to raise. Add tests that call everything and assert little, and the badge climbs. Assertions are expensive to write well, because they require deciding what the correct behaviour is, which is the actual engineering work. So the metric rewards the cheap input and is blind to the expensive one, which is the definition of a metric that gets gamed.

The gaming is rarely malicious, which is what makes it durable. Engineers under a coverage gate write the tests the gate measures, because the gate is what gets reviewed, and the organisation learns to read the number as health. The suite becomes a performance of safety, and performances optimise for the audience.

The incident shape

The bugs that pass covered code have a recognisable shape. The test executes the path but asserts a weak property, an object was returned, a call was made, a status was set, none of which change when the defect is present. Mutation testing makes this visible by construction: mutate the code, and if the suite still passes, the mutation survived, and a surviving mutation is a line that is covered and unchecked. The coverage report said both suites were fine. The mutation report says one of them is theatre.

This is why I treat a surviving mutation as the truer number. It asks the question coverage cannot: would anyone notice if this line were wrong.

The counterargument, fairly

There is a real defence of coverage as a process metric, and I want to state it honestly. As a floor, enforced on new code, it prevents the failure mode of no tests at all, which is common and worse. A team that would otherwise ship untested code, gated to 80 percent, ships tested code, and even weak tests catch some regressions and document intent a little.

Coverage is also the only cheap, comparable, whole repo signal we have. Mutation testing is expensive to run and hard to summarise across a large codebase, so coverage remains the practical proxy for the dark areas map, and proxies earn their keep within their limits.

Both are true. The defect is not in using the metric. It is in promoting it, in letting the badge stand for quality in reviews, in planning and in the sentences managers repeat, when it only stands for reach.

What to use instead, and together

The honest stack is a few signals that cover coverage's blind spot.

Mutation score, on the critical paths at least, because it measures assertion strength directly. Property based tests on the parsers and calculators, because they generate the assertions coverage cannot. And the retrospective measure that needs no tooling: for each recent production bug, name the test that should have caught it, and if the answer is none, the suite's real coverage of what matters is lower than the badge.

Coverage stays in the stack as the flashlight over the dark areas. It just stops being the headline.

It is also worth saying what this does to review culture, because the number shapes behaviour beyond the badge. When coverage is the headline, reviewers stop asking "what does this test assert" because the suite's aggregate number implies the question is settled, and the per test assertion quality, which is the actual property, goes unexamined. Demoting the number restores the question. A reviewer who ignores the badge and reads the assertions is doing the job the badge was pretending to do, and the suite improves because the review measures the thing that catches bugs rather than the thing that climbs.

The rule of thumb

Ask of any coverage number what it is being used to decide. If the answer is "where should we add tests", it is doing its job. If the answer is "is this code safe", it is being asked a question it cannot answer, and the person asking will learn that during an incident, as the quarter's most covered file ships the quarter's worst bug.

Execution is reach. Assertion is judgement. The badge measures the first and markets it as the second, and the difference between them is precisely the bug that passed.

The same dynamic, a number managed upward while the property it proxies stays flat, is the observability argument in observability platforms sold you a search engine, and the test quality version is a test that restates the implementation is not a test.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud