Your Runbook Describes a System You Do Not Have

Every runbook I have ever opened during an incident was wrong. Not catastrophically, not uselessly, but wrong in the specific way that costs fifteen minutes at three in the morning: the dashboard had been renamed, the command needed a flag that did not exist when it was written, and the service it told me to restart had been split into two services eleven months earlier.

Here is my thesis. Runbooks are design documents that got mislabelled as operational documents. They describe the system as it was imagined, they are written once, and nothing in the normal course of engineering ever forces them to be right again.

Why they rot and nothing else does

Code rots too, but code has a forcing function. If it stops working, someone notices immediately and fixes it. A failing test, a broken deploy, a user complaint. The feedback loop is hours long.

A runbook's feedback loop is measured in incidents. If your service has four bad nights a year, a wrong runbook can survive for years and be wrong every single time it is read, because the cost of being wrong is absorbed by whoever is on call rather than by whoever wrote it.

Nothing else in the codebase has that property. Your CI would not tolerate a build script that fails three times out of four and gets fixed by hand each time.

The deeper problem is that runbooks are written by the people with the most context, for the people with the least. The author knows that "restart the ingest worker" means a specific deployment in a specific namespace with a specific gotcha about draining. They write "restart the ingest worker" and move on. Six months later someone who has never seen the system reads those four words literally and makes it worse.

The counterargument, which is fair

There is a real argument that runbooks are worth having even when they are stale, because a partially wrong runbook beats a cold start. The person on call at least knows which subsystem to look at, which metric to check, and who to wake up. That is genuinely valuable and I am not going to pretend otherwise.

I also accept that exhaustive, always current operational documentation is expensive. If you have forty services, keeping forty runbooks tested is a job, and there is a reasonable engineering manager who looks at that cost and decides the money is better spent on making the services simpler.

Both of those are true. And they lead somewhere I did not expect when I started writing this.

What actually works

The teams I have seen get this right do not have better documentation discipline. They have a different forcing function. Three things, in order of how much they helped.

Run the runbook during a game day, on purpose, with someone who did not write it. Not a tabletop exercise where everyone talks about what they would do. An actual drill where a person with no context follows the document literally and you watch where they get stuck. Every stumble is a defect with a stack trace attached. This is the only thing that reliably surfaces the gap, because it reproduces the conditions under which the document will be used.

Delete the parts that are not executable. A runbook that says "check whether the queue is backing up" is prose. A runbook that says "run this command, if the number is above 500 do this" is a procedure. Prose rots because there is no way to be wrong. Procedures break loudly. Most of what I have seen labelled as a runbook should be deleted down to the ten lines that are actually executable, and the rest belongs in an architecture doc where staleness is less dangerous.

Put the runbook next to the thing it describes, and make it fail with it. If the command in your runbook is kubectl rollout restart deploy/ingest and the deployment gets renamed, nothing tells you. But if that command lives in a script in the repository, and the script is exercised by a scheduled job, the rename breaks the job and someone fixes the script. The runbook becomes a pointer to something that is already tested.

The uncomfortable conclusion

Most runbooks are not operational documents. They are a way of feeling prepared. The feeling is real and the preparation is not.

I am not arguing for no documentation. I am arguing that a runbook you have never executed is a hypothesis, and you would not ship a hypothesis about your code.

The rule of thumb I have settled on: if a runbook step has not been executed by someone other than its author in the last six months, treat it as fiction until proven otherwise. Either schedule a drill that proves it, or delete it and write down the one command that matters.

A shorter, tested, partially wrong runbook beats a comprehensive one nobody has ever run. That is not a compromise, it is the actual finding. Every postmortem I have read where the runbook helped was a postmortem where someone had run it before under pressure free conditions, and every one where it hurt was a document that had been written once and left alone.

There is a version of this argument that says the answer is better tooling for keeping docs in sync, and I have tried that too. Link checkers catch dead URLs. They do not catch a command that still runs but now does the wrong thing, or a dashboard that still resolves but no longer shows the metric that matters. The failure mode of a stale runbook is semantic, and semantic drift is only detectable by execution.

The same logic applies to your incident process more broadly, which is the subject of what makes a postmortem useful and the first ten minutes of an incident. Both of those are also documents, and both are also usually fiction.