The Runbook Nobody Reads at Three in the Morning

The fired behaved. The incident behaved. The chain between them is where working as designed stopped meaning working.

This is the thesis: the runbook is written for the reader with time and read by the reader without it, and the gap is the reason the thorough document fails at the moment it exists for.

This is an opinion, and the long document has real value in the wrong moment, and the rule of thumb at the end is where I land.

What the runbook actually is

The runbook is the document, and the document is the thorough, and the thorough is the complete, and the complete is the context, and the context is the architecture and the history and the alternatives, and the three are the useful, and the useful is the afternoon, and the afternoon is the learning, and the learning is the onboarding, and the onboarding is the real audience, and the audience is the not the three in the morning.

The three in the morning is the person with the adrenaline, and the adrenaline is the narrow attention, and the narrow is the three commands, and the commands are the check and the mitigate and the escalate, and the three are the need, and the need is the not the context, and the not context is the gap, and the gap is the unread, and the unread is the incident.

The runbook is the two documents wearing one name, and the two are the reference and the emergency card, and the card is the short, and the short is the readable under stress, and the readable is the used, and the used is the fix, and the fix is the separation, and the separation is the discipline.

Why the long document fails at 3am

The reading is the time, and the time is the absent, and the absent is the scroll, and the scroll is the search, and the search is the keyword, and the keyword is the unknown, because the person does not know the cause yet, and the not know is the reason they are reading, and the reading is the too slow, and the slow is the guess, and the guess is the restart, and the restart is the wrong.

The stress is the working memory's reduction, and the reduction is the paragraph's cost, and the cost is the skip, and the skip is the miss, and the miss is the step, and the step is the fix, and the fix is the missed, and the missed is the incident, and the incident is the reason the emergency card is the numbered list and not the prose.

The document is also the stale, and the stale is the untrusted, and the untrusted is the skip, and the skip is the guess, and the guess is the incident, and the incident is the reason the card should be tested, and the tested is the trusted, and the trusted is the used.

This is the same product framing as the logs that are a product nobody owns, and the shared property is the one worth fixing.

Where the long document earns its place

The document earns it as the reference, and the reference is the afternoon's reading, and the afternoon is the understanding, and the understanding is the incident's review, and the review is the improvement, and the improvement is the value, and the value is the real, and the real is the keep, and the keep is the separate document.

The document earns it as the onboarding, and the onboarding is the new engineer, and the new engineer is the time's having, and the having is the reading, and the reading is the learning, and the learning is the value, and the value is the real, and the real is the keep.

The document does not earn it as the three in the morning's tool, and the not is the gap, and the gap is the card, and the card is the fix, and the fix is the one page, and the page is the discipline.

The rule of thumb

The runbook should be two artefacts: a one-page emergency card with the numbered commands and the escalation rule, and a longer reference for the afternoon. The card is what gets used at three in the morning, and the card should be tested in the game day rather than assumed.

Write the card first. The card is the three commands, and the commands are the check and the mitigate and the escalate, and the three are the one page, and the page is the fix, and the fix is the order, because the card written after the document is the summary and the summary is the still too long.

Put the decision, not the explanation. The card says if this then that, and the that is the command, and the command is the copy-pasteable, and the copy-pasteable is the stress's friend, and the friend is the fix, and the fix is the discipline, because the prose is the afternoon's and the command is the morning's.

Link the reference, do not inline it. The link is the depth's availability, and the availability is the optional, and the optional is the card's shortness, and the shortness is the readable, and the readable is the used, and the used is the fix.

Test the card in a game day. The test is the simulated incident, and the simulated is the reading under pressure, and the pressure is the reality, and the reality is the gap's exposure, and the exposure is the fix, and the fix is the drill, and the drill is the discipline, because the untested card is the assumed card.

Date it and review it with the incident. The review asks whether the card was used and whether it worked, and the ask is the update, and the update is the freshness, and the freshness is the trust, and the trust is the used, and the used is the fix.

Where I land

The thorough runbook is not the failure, the single artefact is, because it tries to serve the afternoon reader and the three in the morning reader with the same document, and the two need opposite things. Write the one-page card first, keep the reference behind a link, and test the card under simulated pressure.

The alert fired at three and the runbook sat unopened in the wiki while the person on call guessed at a restart. Name that one property in review and this class of bug gets hard to ship.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud