A Year of Debugging AI Generated Code

I have been writing about the bugs coding agents produce for about a year. The models have improved substantially in that time, so it seems worth going back and checking which of those observations still hold.

Some of it has aged badly, in a good way. Some of it has not moved at all.

What genuinely improved

Invented APIs are much rarer. A year ago, calling a method that does not exist was a weekly occurrence. Now it is unusual for well known libraries. Better training data and, more importantly, agents that can read the actual package source before writing against it.

The residual case is still the dangerous one: a parameter that is silently accepted and ignored, like passing retries to a function that takes arbitrary keyword arguments. That produces no error and a review comment that reads "we retry three times" when the truth is zero. Strict typing still catches this and nothing else reliably does.

Following existing patterns got much better. Agents now pick up the conventions in a file and match them, including error handling style and test structure. This matters more than it sounds, because it means the output needs less mechanical cleanup.

Verification loops close on their own. The biggest practical change. An agent that runs the tests, reads the failure, and corrects itself handles a category of bug that previously reached review. My review comments are noticeably less about "this does not compile" and more about design.

What did not move

The context problem is structural. Every bug that comes from information not being in the repository is still there, because no amount of model capability fixes an information gap.

Scale is the clearest example. An agent still writes a loop with a database call inside it, because nothing in the code says that list will contain forty thousand items in production. It is not a reasoning failure. The fact is genuinely not available.

Same for concurrency. A read-then-write is correct in a single threaded reading of a file and wrong under two simultaneous requests, and concurrency is invisible in the text.

Error handling is still the modal failure. Catch broadly, log, return null. It remains the most common pattern I remove in review, and I think the reason is that it is genuinely the most common pattern in public code. The training data is the problem and it is not going to change.

Tests derived from implementations are still the most dangerous output. An agent asked to write tests reads the code and asserts what it does. If the code has a bug, the test now certifies it. Green CI around a defect is worse than no tests, because it produces confidence.

This one has arguably got worse in practice, because agents write more tests faster and nobody reviews them as carefully as production code.

Knowing when to stop. An agent asked to fix a failing test will fix it, including by weakening the assertion. It has no sense that some fixes are worse than the bug. I still watch for this in every review.

What I changed in how I work

I write test cases, agents write test scaffolding. The single highest value change. I supply inputs and expected outputs derived from the specification, the agent writes fixtures and parametrisation. Keeps judgement with me and typing with the tool.

AGENTS.md became load bearing. Not the aspirational version with "write clean code", the specific invariants that are invisible in the repository. Money is integer paise. Every handler is concurrent. This table has forty million rows. Those lines measurably change output.

I restart sessions much more aggressively. When an agent makes the same mistake twice, the context is degraded and it will not recover on its own. Two minutes writing a handoff note beats an hour of a session that has lost the thread. The mechanism is worth understanding because it makes the decision obvious rather than feeling like giving up.

I invested in verification infrastructure rather than prompts. Fast tests, strict types, query count assertions, a lint rule for bare exception handlers. These help agents and humans equally, and they work regardless of which model I am using next month.

That last point is the one I would emphasise most. Prompt technique has a shelf life measured in months. A test suite that runs in twelve seconds is still valuable in five years.

What I got wrong

I underestimated how much the tooling matters relative to the model. I spent the first few months thinking about prompts. The variable that actually predicts output quality in my experience is whether the repository has fast, reliable feedback. The same agent in a well instrumented codebase and a poorly instrumented one performs very differently.

I was too pessimistic about agents on unfamiliar code. I assumed they would be worst where context was thinnest. They are frequently very good at orienting in an unfamiliar codebase, because reading and summarising is a strength. It is acting on what they read that needs supervision.

I was too optimistic about review catching things. I assumed a careful reviewer would catch the context bugs. In practice reviewers under-scrutinise clean looking code, and agent code looks clean. Uniform smoothness in a diff touching a gnarly area is now a signal I actively look for.

The thing that has not changed at all

The bottleneck moved from generation to verification, and it has stayed there.

Everything useful I have learned in the last year is downstream of that one observation. The teams doing well with these tools are not the ones with better prompts. They are the ones who noticed that the expensive step is now establishing whether code is correct, and invested accordingly.

Which means the skill that compounds is not prompting. It is the ability to determine whether something works, which is debugging, which is the thing this blog has been about since the first post.

I did not expect writing about debugging to become more relevant because of AI. It has.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud