Why AI Generated Code Passes Tests and Still Breaks Production

A pattern I have run into repeatedly over the past year: a change is generated by an agent, reviewed by a competent engineer, passes every test, deploys, and then does something wrong in production within a week.

The interesting question is not why the code was wrong. Code is wrong all the time. The interesting question is why the safety nets did not catch it, because they usually do for human written code of comparable size.

I think there is a structural answer.

Tests verify consistency, production verifies correspondence

A test suite checks that your code behaves the way your tests say it should. That is a consistency check within a closed system. Production checks whether your code corresponds to reality, meaning real data, real concurrency, real volumes, real users doing things nobody anticipated.

These are different properties, and for human written code they correlate reasonably well, because a human writing a test is drawing on a mental model of the real system. The test encodes something the author knows about the world.

For generated code the correlation is much weaker, because the model's knowledge of your world comes entirely from what is in the repository. Anything true about your production environment but absent from the code is invisible.

So the failure mode is specific. Generated code is highly self consistent and weakly grounded. Tests measure self consistency. That is why they pass.

Where the grounding is missing

Concretely, here is what a repository does not contain.

Scale. Your test fixture has 12 orders. Production has 40,000. Nothing in the code says so. An N+1 query that runs in 40ms in CI takes 90 seconds in production, and the test suite is perfectly happy.

Concurrency. Test suites are usually single threaded and deterministic. A read then write race is functionally invisible until two real users do the same thing within the same 30 milliseconds.

Data messiness. Your fixtures have well formed data because somebody wrote them by hand. Production has the row from the 2019 import with a null in a column that is supposed to be non null, the customer name with an emoji, the address field containing an entire paragraph, the date stored in a different format by a system that was decommissioned three years ago.

Failure of dependencies. In tests, the payment API returns 200. In production it returns 402, times out, returns a 200 with an empty body, or returns success and then reverses the charge asynchronously.

Time. Tests run in seconds. Production runs for months. Memory leaks, unbounded caches, log files, and counters that overflow are all invisible in a test run by construction.

Deployment reality. Multiple instances, rolling deploys where two versions run simultaneously, a cache that survives the deploy, a queue holding messages produced by the previous version.

A human engineer carries all of this implicitly. They have been paged at 3am. That experience is not in the repository and it does not transfer.

The tests are frequently part of the problem

This is the part I find most concerning, because it inverts the safety net.

When you ask an agent to write tests for existing code, the natural way to do it is to read the code and assert what it does. If the code has a bug, the test now asserts that the bug is correct.

def apply_discount(price, discount_pct):
    return price * (1 - discount_pct)

# generated test
def test_apply_discount():
    assert apply_discount(100, 0.2) == 80.0

Reasonable looking. But if the rest of your system passes discount_pct as 20 rather than 0.2, this function is broken and the test permanently certifies it as correct. Worse, when someone later fixes the function, this test fails and looks like a regression, so there is a decent chance they change the fix rather than the test.

The general principle: a test derived from the implementation cannot detect a bug in the implementation. It can only detect a change. Those are very different guarantees and the green checkmark looks identical.

The mocking version is subtler:

@patch("app.billing.charge")
def test_checkout_success(mock_charge):
    mock_charge.return_value = {"ok": True}
    result = checkout(cart)
    assert result.status == "success"

This test asserts that checkout succeeds when charge succeeds. It contains no information. It will pass if the real charge starts returning a different shape, if it starts throwing, if the argument order changes, or if checkout stops calling it. Every meaningful failure mode is mocked out.

I have seen suites with excellent coverage numbers made almost entirely of this pattern. Coverage measures which lines executed, not whether anything was verified.

Why review misses it too

Human review should catch what tests miss, and for generated code it underperforms in a specific way.

The code reads well. It is well named, evenly structured, consistently formatted, and it follows the conventions of the surrounding file. Reviewers are trained, largely unconsciously, to allocate scrutiny to code that looks messy. Uniformly clean code gets less attention, and generated code is uniformly clean.

I have started treating this as a signal in itself. If a diff touching a genuinely gnarly part of the system comes back tidy and symmetrical, the gnarl was probably ignored rather than resolved. Real code that has survived production usually has a weird branch with a comment explaining an incident.

The volume is wrong for careful review. Agents produce large diffs quickly. A 900 line pull request gets a different quality of attention than a 90 line one, regardless of who wrote it. Research on review effectiveness has been consistent about this for years, and generation makes large diffs cheap to produce and no cheaper to review.

Reviewers check the code against itself. The natural review question is "is this correct?" For internally consistent code the answer is usually yes. The question that catches these bugs is "does this do what we asked, given things about our system that are not visible in this file?" That requires holding the requirement and the production context in your head simultaneously, which is much harder work.

What actually helps

I do not think the answer is to stop using these tools. The productivity is real. But the verification has to change shape.

Write the test cases yourself, let the agent write the scaffolding. This is the single highest value change. You supply the inputs and expected outputs, derived from the specification or from your knowledge of the domain. The agent writes the boilerplate, the fixtures, the parametrisation. Now the assertions are grounded in something outside the implementation.

Test against real data volumes. Seed your integration test database with a production sized dataset, or at least a production shaped one. An N+1 that is invisible with 12 rows is obvious with 40,000, and this catches an entire category automatically.

Assert on query counts. A test that says this endpoint issues at most four queries catches N+1 patterns mechanically, with no reviewer judgement required. Cheap to add and it has caught more real problems for me than most other assertions.

Put your invariants in AGENTS.md. The constraints that live in your head and not in the repo: never catch bare exceptions, config must fail fast at startup, assume every handler runs concurrently, no ORM calls inside loops, all money is integers in the smallest unit. Writing these down converts a recurring review comment into a non event, and it is the highest leverage thing I have found for improving generated output quality.

Use property based testing where it fits. Hypothesis or fast-check generate inputs you would not have thought of, which is precisely the gap. A property like "the discount is never greater than the price" catches the 0.2 versus 20 bug immediately, and it is one line.

Make it mechanical wherever possible. Strict types catch invented APIs. Lint rules catch swallowed exceptions and permissive defaults. Unique constraints catch check then act races by turning them into loud errors. Reviewer attention is scarce and should be spent on the things only a human notices.

Review against the requirement, not the diff. Open the ticket next to the code. The question is whether this does what was asked, not whether it is internally coherent.

Deploy behind flags and watch. Since some of this genuinely cannot be caught before production, shorten the feedback loop. Ship to a small percentage, watch error rates and latency, and expand. Treat generated changes to critical paths with the same caution you would give a change from a new team member, because in an important sense that is what it is.

The uncomfortable framing

There is a version of this post that says AI code is bad. That is not my view and it does not match my experience. I use these tools daily and they have made me substantially faster.

The accurate framing is that we have made one part of the process dramatically cheaper without changing the part that validates it. Generation went from hours to minutes. Verification did not move at all. And our verification tools were calibrated for a world where writing code was expensive, which meant the volume of code needing verification was naturally limited by how fast humans could type.

That constraint is gone, and it was doing more work than we realised.

The teams handling this well are not the ones using the tools least. They are the ones that noticed the bottleneck moved and invested in the new bottleneck: better integration tests, realistic data, mechanical checks, explicit invariants, faster production feedback.

Which is, I think, the actual lesson. The skill that matters now is not writing code and it is not prompting. It is being good at establishing whether code is correct, which is a debugging skill.