Your Average Latency Is Fine and Your Users Are Not
A service I worked on had a mean response time of 84 milliseconds. The dashboard was green for months. Then someone plotted the distribution and we found that one request in fifty took over two seconds, and those users were bouncing.
The average was not lying. It was just describing a user who does not exist.
Why the mean is structurally useless for latency
Latency distributions are not normal. They are long tailed and usually close to log normal, which means the mean sits somewhere in the bulk of the distribution while the interesting behaviour lives far to the right.
A concrete example. Nine hundred requests at 80ms and one hundred requests at 900ms gives a mean of 162ms. That number looks acceptable. But ten percent of your users are having an experience five times worse than the headline figure suggests, and if those are the users hitting an uncached path, they are the ones doing the most work.
The mean is also unstable in the wrong direction. A single 30 second timeout in a sample of a thousand moves the average by thirty milliseconds, which is invisible. It does not tell you that a user waited half a minute.
The percentiles that mean something
| Percentile | What it tells you | Who it describes |
|---|---|---|
| p50 | The typical request | Your median user, one request |
| p90 | Where the bulk ends | Users on a slightly worse path |
| p99 | The tail begins | One in a hundred requests, every heavy user |
| p99.9 | Your worst real experience | Your most valuable and most active users |
The reason p99 matters more than intuition suggests is that your heaviest users experience the tail constantly. A user who makes ten requests a day hits your p99 roughly once a week. A user who makes five hundred requests a day hits it five times a day. Your power users live permanently in the part of the distribution your average hides.
This is why "our p50 improved" is almost never the interesting result, and why comparing two systems on mean latency will routinely pick the wrong one.
Retries multiply the tail
Here is the part that turns a mild tail into an outage. If one backend has a 1% chance of a slow response and your request fans out to five backends, the probability that at least one is slow is:
1 - (0.99)^5 = 4.9%
Fan out to twenty and it is 18%. The tail of your composite request is much worse than the tail of any component, and no amount of individual component tuning fixes it.
Now add a retry. If you retry the slow one, you have made the tail worse before you made it better, because the retry lands during the same period of contention that caused the original slowness. This is the mechanism behind retry storms, and it is why retry logic needs a budget and jitter rather than a well intentioned loop.
What actually causes the tail
In my experience the tail is almost never your main code path. It is one of these.
Cache misses. The p50 is a cache hit. The p99 is a cold read from the database. If your hit rate is 95%, your p95 and above is entirely determined by the miss path, and optimising the hit path changes nothing above the median.
Garbage collection. A stop the world pause of 200ms in a JVM or a major GC in Go will show up as a spike on exactly the requests that were in flight. It is invisible in the mean and unmistakable in p99.9.
Connection pool waits. Time spent waiting for a connection is latency, and it is bursty. A pool at 90% utilisation has a small mean wait and an enormous tail wait. This is queueing theory, not bad luck.
Noisy neighbours and CPU throttling. A container that hits its CFS quota gets throttled for the rest of the 100ms period, which produces exactly the shape of a long right tail. Covered in container CPU throttling.
Lock contention. Under low load a mutex is free. Under high load the same mutex produces a distribution where most threads acquire immediately and a few wait for everything ahead of them.
Background work colliding with requests. Autovacuum, a log rotation, a metrics scrape, a cron job. All periodic, all producing a tail at a predictable interval. If your p99 spikes every five minutes, look for something that runs every five minutes.
How to measure it properly
Percentiles do not average. This is the most common measurement error I see. You cannot take the p99 from ten instances and average them to get a system p99. You need the raw observations, or a data structure that merges.
The practical options:
Histograms. Bucket the observations and compute percentiles from the buckets. Prometheus histograms do this, and histogram_quantile() merges correctly across instances. The cost is bucket boundary choice, and boundaries that are too coarse give you a p99 that is really a p95.
t-digest or HDR histogram. Mergeable summaries with good accuracy in the tail. More machinery, better numbers.
Raw sampling. Keep every observation for a short window. Expensive, exact, and worth it when you are actively hunting a tail problem.
What you must not do is export a per instance p99 and average it, or export a mean and a max. The max is a single outlier and the mean is the number that got you here.
What I put on a dashboard
Three lines, and only three: p50, p99, and request rate. The rate matters because a p99 measured over ten requests is noise, and a p99 measured over ten thousand requests is a fact.
Then one alert, on p99 over a five minute window, with a threshold set from observed behaviour rather than from a round number. And I look at the shape of the distribution during incidents, not the value, because a tail that has moved right is a different problem from a bulk that has moved right, and they have completely different causes.
If you take one thing from this: stop optimising the mean. Nobody experiences your mean. They experience one draw from your distribution, and the draws that make people write angry emails are all in the tail. Three metrics that catch incidents covers the rest of the set I actually watch.
