The Model Should Not Write the Tests for the Code It Wrote

I started at the reason the suite passed on code that was wrong, because that is where the symptom landed, and walked it backwards link by link until the green stopped looking innocent.

This is the thesis: a model should not write the tests for the code it wrote, and the reason is not that the model writes bad tests, it is that the tests are derived from the implementation, and a test derived from the implementation cannot detect an error in the implementation, because the error is in the derivation, and the derivation is shared.

This is an opinion, and the opposing view is reasonable, and the rule of thumb at the end is where I land.

The derivation is the problem

A test's value is its independence from the implementation, because the test is the specification made executable, and the specification is the intent, and the intent is what the implementation should satisfy, and the test checks the satisfaction, and the check is meaningful only if the test was written against the intent rather than the implementation, because a test written against the implementation asserts that the implementation does what it does, which is a tautology, and the tautology always passes, and the always passing is the uselessness.

When a model writes the code and then the tests, the tests are generated from the code, because the code is the context, and the context is the source, and the source is the implementation, so the tests assert the implementation's behaviour, and the behaviour includes the errors, and the errors are asserted as correct, and the assertion is green, and the green is the tautology, and the tautology is the suite, and the suite is the signal everyone trusts.

The model is not being deceptive. The model is doing what it was asked, which is write tests for this code, and the tests for this code are the tests that pass on this code, and the passing is the requirement, and the requirement is satisfied, and the satisfaction is the problem, because the requirement was wrong, and the wrong requirement is the one that asked the implementation's author to write the implementation's tests.

Why the green suite is worse than no suite

A missing suite is a visible gap, and the known gap is compensated for by other means: the manual testing, the staging, the canary, the caution. A green suite is a filled gap, and the filled gap removes the caution, so the risk becomes unmanaged, and the unmanaged risk is the incident. The incident is worse than the one the missing suite would have produced, because the missing suite kept the caution and the green suite removed it.

This is the same misplacement as agents should not merge their own pull requests, which is why the two fixes look alike.

The opposing view, and where it is right

The opposing view is that model-written tests are better than no tests, and the view is correct in the case where the alternative is none, because a tautological suite still catches regressions, and the regression is a real class of failure, and the catching is real value, and the value is greater than zero, and zero is the alternative. The view is also correct that the tests document the behaviour, and the documentation is real value, because the behaviour is written down, and the written behaviour is reviewable, and the reviewable behaviour is the entry point for the human to notice the error, and the noticing is the fix.

Where the view fails is the claim that the green suite is evidence of correctness, because the suite is derived from the code, and the derivation means the suite and the code share their errors, and the shared errors are the ones the suite cannot catch, and the uncatchable errors are the ones that matter, because the catchable ones would have been caught by the code's own tests anyway, and the uncatchable ones are the specification violations, and the specification violations are the incidents.

The opposing view is right about the value and wrong about the evidence, and the distinction is the whole argument, because the value is real and the evidence is not, and the confusion of the two is what makes the green suite dangerous.

The thumb rule I use

The tests should come from a different source than the code, and the source can be a human, a different model, a different prompt, or the specification, and the requirement is independence, and the independence is what makes the test a check rather than a restatement.

Write the tests from the specification, not the code. The specification is the intent, and the tests from the intent check the implementation against the intent, and the check is meaningful, because the test and the code have different sources, and the different sources mean the errors are not shared, and the unshared errors are the catchable ones, and the catching is the value.

Generate the tests before the code. The order matters, because the tests written first are written against the specification, and the specification is the only source available, and the code written after must satisfy the tests, and the satisfaction is the implementation, and the implementation is checked by tests it did not generate, and the checking is the independence.

Use a different model or a different prompt for the tests. The independence can be approximated by the separation, and the separation is imperfect but better than none, because the different model has a different distribution, and the different distribution means the errors are less correlated, and the less correlated errors are more catchable, and the catching is the value, and the value is proportional to the separation.

Have the human own the specification. The specification is the human's, and the tests are derived from the human's specification, and the derivation is the independence, and the independence is the control, and the control is the human in the loop, and the loop is the fix, because the human is the only source that is not the model.

Where I land

The model can write the tests, and the tests have value, because the regression coverage and the documentation are both real. The model should not be the only source, and the green suite should not be read as evidence of correctness, because the suite is derived from the code and the derivation is the shared error.

The suite was green and the code was wrong, and the green was the model confirming the model. Name that one property in review and this class of bug gets hard to ship.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud