Debugging Production Without Touching Production

The worst bugs are the ones that only exist in production. Not because production is special, but because production has history, and your local environment has none.

A staging database restored from last night's backup is not the same thing. It has the rows and none of the circumstances: the connection pool that has been open for nine days, the log file that filled the disk to 94 percent, the cache warmed by a traffic pattern you cannot replay, the config change someone applied by hand in March and never committed.

I spent a long time trying to reconstruct those conditions and eventually concluded that reconstruction is the wrong approach.

Why a database dump is not enough

The standard reproduction attempt goes: dump the database, restore locally, run the code, watch the bug not happen. Then you start listing what is different, and the list is longer than you expect.

Time dependent state. A cache with entries at various ages. Sessions at various points in their lifetime. A job queue with items that have been retried four times. None of this is in a dump.

Filesystem accumulation. Temporary files, rotated logs, a /tmp with things in it, a disk that is nearly full. Disk pressure changes behaviour in ways that are extremely hard to simulate deliberately.

Process state. A connection pool that has been running for a week has connections in states a fresh pool does not. Memory is fragmented. A long lived process has a heap shaped by its history.

Configuration drift. The gap between what is in your repository and what is actually on the machine. Everybody has some. Nobody knows how much.

Kernel and package versions. Production is on the versions it was built with. Your laptop is on whatever brew upgrade last decided.

Any one of these can be the reason the bug does not reproduce. Collectively they explain most of the "works locally" frustration, and it is not really a code problem at all.

Cloning the machine instead of the data

The alternative: snapshot the running machine, restore it somewhere isolated, and debug the copy.

krova snapshots create --cube prod-api-01 --name incident-4471
krova cubes create debug-4471 --snapshot incident-4471

The clone has the same disk, the same files, the same configuration drift, the same nearly full log directory. Everything on disk at the moment of capture.

What is worth being precise about is the boundary between what survives and what does not, because being vague about this leads people to expect too much.

Disk snapshots restore the filesystem, not the running processes. When the clone boots, it boots fresh and the services start from their init scripts against the captured disk. Your nine day old connection pool is gone. What remains is everything that was written down, which in my experience covers the large majority of hard reproduction cases.

The kernel row deserves a note: on a platform where the kernel is supplied by the host rather than living on the guest disk, a restored snapshot gets the host's current kernel rather than the one that was running when you captured. That surprised me once, and I wrote it up in the reboot that did not change my kernel.

How the storage works, and why it is cheap

The reason this is practical rather than merely possible is content addressed deduplication. On Krova Cloud snapshots use Restic, which splits the disk into variable sized chunks, hashes each one, and stores each unique hash once.

The consequences matter for how you use it:

The first snapshot is roughly the size of the used disk. A 40 GB volume with 12 GB written stores about 12 GB, before compression.

Every subsequent snapshot stores only changed chunks. A daily snapshot of a machine where 200 MB changed costs 200 MB, not 12 GB.

Ten clones of one snapshot share the same underlying chunks. Restoring is a read of the chunk store rather than a copy of the whole disk, which is why you can spin up several at once without waiting.

It is encrypted at rest, which matters given a production snapshot contains production data, including whatever secrets are on the disk. That last point is not a footnote, and I come back to it below.

Practically, snapshotting a machine before any risky operation becomes free enough that you stop thinking about it, which is the real behavioural change.

Debugging the clone

The clone is disposable, and that changes what you are allowed to do.

Attach a debugger and stop the world. gdb, py-spy dump, rbspy, a JVM heap dump. On production these are somewhere between rude and outage inducing. On a clone, pause anything for as long as you like.

Run strace on everything. The overhead does not matter. Trace the whole process tree if it helps.

strace -f -tt -T -p $(pgrep -f api-server) -o /tmp/full.trace

Delete things to test hypotheses. Think the corrupted cache file is the cause? Delete it and restart. If you are wrong, destroy the clone and restore the snapshot again. This is the single biggest practical difference: hypothesis testing becomes cheap because being wrong costs nothing.

Run the destructive migration. The one you are not sure about. Run it, watch it fail, fix it, restore, run it again. Then run it on production with actual confidence rather than hope.

Bisect against real state. Deploy successive commits onto the clone. A bisect that needs production data to reproduce is otherwise nearly impossible.

The rules I follow

Treat the clone as production data, because it is. Same access controls, same retention policy, same people. A debug machine holding a copy of your customer database is a production system wearing a false moustache. Delete it when you are finished and set an expiry so that forgetting is not fatal.

Snapshot at the moment of the incident, not afterwards. State that explains a failure is often the first thing cleaned up by a restart or a log rotation. Capturing costs seconds. Make it the first step of incident response, before mitigation, because mitigation frequently destroys the evidence. This is the same instinct as capturing state before restarting anything.

Name snapshots after the incident. incident-4471 tells you why it exists. snapshot-2026-08-24-3 tells you nothing, and in three weeks nobody will know whether it is safe to delete.

Rotate any credentials that were on the disk if the clone was shared more widely than production access normally allows.

Destroy it. Per minute billing means an idle clone is cheap, not free, and the security exposure grows with every day it exists.

The habit worth forming is smaller than it sounds. Before you touch anything during an incident, take a snapshot. It takes seconds, it costs almost nothing, and it converts a one shot investigation with real consequences into one you can repeat as many times as you need.