What to Do in the First Ten Minutes of an Incident

Disclosure: I have been on call for shared hosting, for a theme platform, and now for infrastructure. This is what I wish somebody had told me in 2019.

The page goes off. Something is broken. The instinct, especially early in your career, is to open a terminal and start looking.

That instinct is wrong, and specifically it is wrong for the first ten minutes. Not because investigating is bad, but because four other things have a higher return and they make the investigation faster when you get to it.

1. Establish what is actually broken, from the outside

Before looking at any internal dashboard, answer: what can a user not do right now?

curl -w "%{http_code} %{time_total}s\n" -o /dev/null -s https://yourapp.com/api/health

Load the product. Try the main flow. Check from outside your network, because a VPN or an internal DNS entry can make a broken thing look fine.

This matters for two reasons. It tells you the blast radius, which determines how you respond. And it gives you a test you can rerun to know when you have fixed it, which you will need later and which is much harder to construct once you are deep in a hypothesis.

I have watched people spend forty minutes on a subsystem that turned out not to be user visible, because they started from an alert rather than from the product.

2. Say something

One message, in the channel people are watching:

Investigating elevated errors on checkout. Started ~09:14.
Impact: some customers cannot complete orders. Cause unknown.
I'm on it. Next update 09:35.

Four things: what, since when, impact, and when you will next speak. That last one is the important one, because it stops people asking and lets them plan.

The cost of not doing this is that you get interrupted every four minutes by someone asking for a status, and each interruption costs you the context you were holding.

If it is a customer facing outage, get the status page updated. Somebody other than you should own that.

3. Ask what changed

Before forming a hypothesis about the code, check whether anything moved.

• Deploys in the last few hours, across all services and teams, not just yours
• Config changes and feature flag flips
• Infrastructure changes, node pool updates, certificate rotations
• Provider status pages for anything you depend on
• Scheduled jobs that run on this cadence

Most incidents follow a change. Finding the change is frequently faster than diagnosing the symptom, and it gives you a candidate fix immediately: revert it.

The mistake I have made repeatedly is scoping this to my own service. I once lost an hour on a DNS latency problem because I searched for deploys to the alerting service, found none, and concluded nothing had changed. The trigger was another team's rollout seven minutes earlier.

"What changed" means everything in the blast radius, not everything you own.

4. Decide whether to mitigate or investigate

The most important call, and the one most often made implicitly.

Mitigation restores service. Revert the deploy, flip the flag off, scale up, fail over, disable the feature.

Investigation understands the cause.

They are different jobs and mitigation comes first when users are affected. You can investigate a reverted deploy at leisure. You cannot investigate calmly while checkout is down.

The trap for engineers is that investigation is more interesting. Rolling back feels like giving up, and understanding feels like the real work. During an outage that preference costs somebody else money.

The question that resolves it: is there an action I can take right now that probably restores service, without needing to know the cause? If yes, take it.

Reverting a deploy that correlates in time is a good bet even without proof. If it does not help, you have eliminated a hypothesis cheaply.

Two caveats. A revert that runs a database migration backwards may not be safe, so know which of your deploys are reversible before you need to. And if the mitigation destroys evidence, capture it first: a heap dump, a goroutine dump, the current logs, a snapshot of the metrics.

Then investigate, in a specific order

Once service is restored or you have decided mitigation is not available:

Read the shape of the symptom. p99 up with p50 flat means a conditional slow path. Both up means uniform degradation. Errors on one endpoint means that code path. Errors everywhere means something shared.

CPU flat with latency up means waiting, not computing. That is a lock, disk, or network, and it rules out a whole category immediately.

Check the shared infrastructure if one service is slow and its dependencies all report healthy. DNS, the service mesh, the CNI, the connection pool, conntrack. These are shared, rarely instrumented, and they produce exactly that signature.

Look at the population, not the instance. If it is intermittent, aggregate. Which users, which region, which instance, which time of day. The pattern is usually visible in ninety seconds of aggregation and invisible in any single failure.

Things I have learned the hard way

Write down what you try, as you try it. In the incident channel. Two hours in you will not remember whether you already checked something, and the person who joins at hour two needs it. It also becomes the timeline for the postmortem, and a timeline with the wrong hypotheses in it is the valuable kind.

Change one thing at a time. Under pressure the instinct is to change three things and see if it improves. Then you do not know which worked, and one of them may have made things worse in a way you will discover later.

Get a second person for anything longer than thirty minutes. Not to work in parallel, to be the person who asks "have you checked whether it is DNS". Fresh eyes catch the assumption you have stopped questioning, which is the whole game in debugging.

Hand over if you are tired. Decision quality at hour four is meaningfully worse than at hour one, and you will not notice. A handover with good notes beats a tired person continuing.

Do not skip the "is it just me" check. Your VPN, your DNS cache, your stale auth token. It costs ten seconds and I have seen thirty minutes lost to a local problem more than once.

What I would not do

Do not start by reading code. The code has not changed unless something deployed. Start with what changed and what the system is doing now.

Do not tune configuration hoping it helps. Raising a timeout, adding replicas, or bumping a connection pool without a hypothesis is guessing, and it obscures the signal.

Do not restart everything. It occasionally works and it destroys the evidence, which means the next occurrence starts from zero. If you must restart, take a diagnostic snapshot first.

Do not promise a fix time. Promise the next update time. You do not know when it will be fixed and saying otherwise sets up a second failure.

The summary

First ten minutes: confirm impact from outside, communicate, find what changed, decide whether to mitigate.

Then investigate, from the shape of the symptom outward, one change at a time, writing it down.

None of that is clever. It is just the order that consistently produces shorter incidents, and the order I did not follow for the first two years I was on call.