Most Postmortems Are Theatre and Here Is What Makes One Useful

I have written and read a lot of postmortems. Most of them were a waste of an afternoon.

They followed the template. Timeline, root cause, impact, action items. They were filed. They were never read again. Several of the action items were still open when a similar incident happened eight months later, and nobody connected the two.

The format is not the problem. The problem is that most postmortems are written to demonstrate that an incident was taken seriously rather than to change what happens next.

The tells of a useless one

A single root cause. Real incidents have a chain: a change, a gap in testing that let it through, a monitoring gap that delayed detection, and something about the system that turned a small failure into a large one. "The root cause was a bad config value" describes the trigger and skips everything that made the trigger matter.

Action items assigned to nobody, or to a team. "Improve monitoring, owner: Platform" is not an action item. It is a wish. Nothing without a named person and a date will happen.

Human error as an explanation. "The engineer ran the wrong command" is where the analysis stops in a bad postmortem and where it starts in a good one. Why was the wrong command easy to run? Why did nothing catch it? Why was there no confirmation on a destructive operation?

No mention of what went right. Detection worked, or a circuit breaker fired, or somebody's guard rail limited the damage. Those are the things you want more of, and if you never name them they get removed in the next refactor by someone who does not know why they exist.

Written by one person and reviewed by nobody. The value is largely in the conversation.

What a useful one contains

A timeline with the decision points

Not just what happened, but what people believed at each moment.

09:14  Alert: p99 latency on checkout-api above 500ms
09:16  On-call acknowledged. Checked database dashboards, all normal.
09:22  Hypothesis: payment provider slow. Checked their status page, green.
09:31  Hypothesis: our own deploy. Checked, nothing shipped in 4 days.
09:48  Noticed CPU flat while latency high. Concluded time is spent waiting.
10:05  strace on one pod showed 5s timeouts on port 53.
10:09  Scaled CoreDNS from 2 to 6 replicas.
10:11  p99 recovered.

The wrong hypotheses matter more than the right one. They tell you what information was missing and what a reasonable person concluded from what was available. That is the actionable part, because next time you can make that information available faster.

A timeline of only correct steps is a story about how clever everyone was. Nobody learns from it.

The contributing factors, plural

Instead of one root cause, list every condition that had to be true.

For the incident above:

• Another team's rollout increased DNS query volume seven minutes earlier
• Cluster ndots:5 amplified external hostname lookups fivefold
• CoreDNS was running two replicas with no autoscaling
• Nothing monitored CoreDNS at all
• DNS resolution was not instrumented in application traces
• Change correlation tooling filtered to the alerting service by default, hiding the trigger

Remove any one of those and the incident is smaller or shorter. That is six places to intervene rather than one, and it lets you pick the cheapest effective one rather than the most obvious.

I wrote up that DNS incident in detail, and the most useful output was not the fix, it was noticing that the deploy correlation tool had a default that made the trigger invisible.

The detection and diagnosis gap

Two numbers worth tracking separately:

Time to detect. From the problem starting to somebody knowing. If this is large, the gap is monitoring.

Time to diagnose. From knowing to understanding. If this is large, the gap is observability or documentation.

Time to mitigate. From understanding to service restored.

These have different fixes and lumping them into one duration hides which one you should work on. An incident with 2 minutes to detect and 90 minutes to diagnose is an observability problem. The reverse is a monitoring problem.

Action items with names, dates, and sizes

[ ] Add CoreDNS request duration and error rate to the platform dashboard
    Owner: Priya   Due: 12 Jun   Est: 2h

[ ] Set ndots:2 on services that primarily call external hosts
    Owner: Rohit   Due: 19 Jun   Est: 1d

[ ] Deploy NodeLocal DNSCache
    Owner: Priya   Due: 3 Jul    Est: 3d

[ ] Change deploy correlation default to show all services
    Owner: Sam     Due: 12 Jun   Est: 30m

An estimate is what makes these real. An action item with no size is one somebody will keep deferring, and a 30 minute item that prevents an hour of confusion is the highest return work in the list.

Fewer, done, beats more, filed. Three items completed within a month is a better outcome than twelve that are still open next quarter. If the list is long, be honest about which ones you are actually going to do and delete the rest.

Blameless does not mean vague

"Blameless postmortem" gets misread as not naming what happened. That produces documents so hedged they contain no information.

The useful version: name the actions precisely, do not attribute them to character.

"The deploy skipped the staging environment because the pipeline allows a manual override and there was time pressure" is precise and blameless. It names the action, the mechanism that permitted it, and the pressure that motivated it.

"Someone was careless" is neither precise nor useful. "A mistake was made" is neither precise nor blameless, it is just cowardly.

The test I apply: could a reader who was not there reconstruct what happened and why it seemed reasonable at the time? If the answer is no because the document is too vague to be uncomfortable, it has failed.

Run the meeting differently

The document is the artifact. The conversation is where the value is.

Send the draft before the meeting. Reading it aloud wastes the time of everyone in the room.

Spend the meeting on the contributing factors, not the timeline. The timeline is settled fact. The interesting disagreement is about which factors matter and what to do.

Invite people who were not involved. They ask the questions insiders have stopped asking. "Why does that service need write access at all?" is the kind of question that only comes from outside.

Ask what would have made this a non-event. Not what would have prevented it, which invites "be more careful". What structural property would have made it small: a limit, a confirmation, a circuit breaker, a lower blast radius.

The part everyone skips

Follow up.

A postmortem process with no mechanism to check whether action items were completed is a documentation exercise. Put the items in your normal work tracker, not in the document, and review open incident actions at whatever cadence you review other work.

The single best indicator of whether a team's postmortem process is real: pick an incident from six months ago and check whether its action items are done.

If they are, the process works. If several are open and nobody noticed, you are producing documents rather than changing systems, and the next similar incident will get a similar document.

Which incidents deserve this

Not all of them. A postmortem for every blip trains people to write them carelessly.

I would write one when: customers were affected for a meaningful duration, the same class of thing has happened before, diagnosis took much longer than expected, or the incident revealed something surprising about the system.

That last one is underrated. An incident with no customer impact that showed you something you did not know about how your system behaves is worth writing up precisely because it was cheap. You got the information without paying full price for it, which is the reason a loud bug is a gift.