Three Metrics That Would Have Caught Most of My Production Incidents

Disclosure: I run an infrastructure company, so I spend a lot of time thinking about this. Nothing here requires buying anything.

I went back through the incidents I have been involved in over the last few years, both at a hosting company and building products, and asked a narrow question: what single metric would have given the earliest useful warning?

The answers clustered much more than I expected. Three signals cover most of it, and none of them are the ones people set up first.

1. Saturation, not utilisation

The most valuable and the most commonly missing.

Utilisation is how much of a resource is in use. Saturation is how much work is queued waiting for that resource. They diverge in an important way: utilisation caps at 100 percent and stops telling you anything, while saturation keeps growing and tells you how bad it is getting.

A CPU at 100 percent could mean comfortably busy or catastrophically overloaded. Run queue length distinguishes them.

The saturation metrics worth having:

On modern Linux, pressure stall information is the best single source and it is criminally underused:

cat /proc/pressure/cpu
cat /proc/pressure/memory
cat /proc/pressure/io

some avg10=12.43 avg60=8.21 avg300=3.11 total=91827364
full avg10=2.10  avg60=1.44 avg300=0.52 total=12093847

some is the percentage of time at least one task was stalled waiting for that resource. full is the percentage where everything was stalled. Anything sustained above zero on full means real contention.

PSI gives you warning in a way utilisation does not, because pressure rises before throughput collapses. I have watched a service look completely healthy on CPU utilisation while some avg10 for memory sat at 40 percent, which was the disk thrashing that eventually took it down.

The connection pool one is worth calling out separately. Most pool libraries expose a wait count and nobody graphs it. A pool that is fully checked out with requests queueing behind it produces latency that looks like the database is slow, when the database is fine and your pool is too small or your transactions are held too long.

2. Error budget burn rate, not error count

Error count alerting produces two failure modes: it is too noisy at low thresholds and too slow at high ones.

Burn rate fixes both. Define an objective, say 99.9 percent of requests succeed over 30 days. That gives you an error budget of 0.1 percent. Burn rate is how fast you are consuming it relative to the rate that would exactly exhaust it over the window.

Burn rate 1 means you will use exactly your budget by the end of the period. Burn rate 14.4 means you will burn the entire month's budget in about two days.

The practical setup is multi window:

Page:   burn rate > 14.4 over 1 hour   AND  > 14.4 over 5 minutes
Ticket: burn rate > 6    over 6 hours  AND  > 6    over 30 minutes

The short window in each pair stops you paging on something that has already recovered. The long window stops you paging on a two minute blip.

What makes this better than a threshold on error rate is that it is proportional to consequence. A one percent error rate on an endpoint serving ten requests a minute is not the same incident as one percent on an endpoint serving fifty thousand, and a raw threshold treats them identically.

The other benefit is social. "We have used 60 percent of the month's error budget" is a conversation a team can have about priorities. "Errors were elevated on Tuesday" is not.

3. The absence of expected events

The one nobody sets up, and the one that caught the most surprising failures.

Almost all monitoring alerts on the presence of something bad. A large category of failures produces nothing at all:

• A cron job that stopped firing four months ago
• A queue consumer that died and left messages accumulating
• A backup that ran, succeeded, and produced an unrestorable archive
• A webhook the provider stopped sending after a config change
• A scheduled export whose credentials expired

In every one of those the error rate is zero, the CPU is fine, and the dashboards are green. The system is quietly not doing its job.

The pattern is a dead man's switch: the job reports success, and you alert when the report does not arrive.

# at the end of a successful run
requests.post(f"https://monitor.example.com/ping/{JOB_ID}", timeout=5)

Then configure the monitor to alert if no ping arrives within the expected interval plus a grace period. Healthchecks.io, Cronitor, and Prometheus's absent_over_time all do this. The Prometheus version:

absent_over_time(job_last_success_timestamp[2h])

Related and equally useful: alert on queue depth trending up rather than on consumer errors. A consumer that is silently doing nothing has no errors, and the queue depth graph is the only place it shows.

And for anything with a backup: alert on restore success, not backup success. A backup job that completes is not evidence of anything. A periodic automated restore into a scratch environment is. This is the single most commonly skipped control I have encountered, and the failure mode is discovering it during the incident where you needed the backup.

What I would set up first on a new service

In order, because ordering matters when you have limited time:

1. Is it up. A synthetic request from outside your infrastructure, hitting a real endpoint. Not a health check that returns 200 unconditionally.
2. Latency histogram, not average. p50, p95, p99. An average hides everything; a p99 rising while p50 is flat is a completely different problem from both rising together, and the average cannot distinguish them.
3. Error budget burn rate on the two or three endpoints that matter.
4. Saturation on your constrained resource, which is usually the connection pool or memory.
5. Dead man's switch on every scheduled job.
6. Structured logs with a request id, so you can go from a slow trace to the exact log lines.

That is maybe half a day of work and it covers most of what I have needed during incidents.

What I would deprioritise

Dashboards with forty panels. During an incident nobody reads them. Three panels that answer "is it up, is it slow, is it erroring" get used.

Alerting on individual resource thresholds. CPU above 80 percent is not an incident. It is a fact. Alert on symptoms users experience, and use resource metrics for diagnosis after the alert fires.

Tracing before logging. Distributed tracing is excellent and it is a bigger investment than structured logs with correlation ids. Do the cheap thing first.

Perfect instrumentation coverage. The instrumentation you need is the instrumentation for the incident you have not had yet, which you cannot predict. Add it after each incident instead. The DNS problem I mentioned above went undetected because nothing timed name resolution, and the fix was one span added afterwards.

The uncomfortable one

The thing that would have prevented the most incidents is not a metric.

It is knowing what changed. Most incidents follow a change: a deploy, a config edit, a dependency update, a certificate rotation, a provider's maintenance. An annotated timeline of every change across your whole system, overlaid on your latency and error graphs, resolves a large fraction of investigations in the first minute.

Most teams have this data scattered across CI logs, a Slack channel, and someone's memory. Getting it onto one timeline is unglamorous plumbing and it outperforms a lot of more sophisticated observability work.

I lost an hour of an investigation once because I searched for deploys to my own service, found none, and concluded nothing had changed. The trigger was another team's rollout seven minutes earlier. The data existed. It was just not anywhere I would look.