Why LLM Latency Is Bimodal and the Average Hides It

I started at the number nobody experienced, because that is where the symptom landed, and walked it backwards link by link until the four seconds stopped looking innocent.

This is the anatomy of the bimodal latency distribution, and of the specific property that makes the average actively misleading: the mean of two separated clusters lands in the gap between them, and the gap is empty, so the reported number describes a request that never happened.

This applies to any feature calling a hosted model, and the mechanism is the queueing at the provider, which is the same regardless of the vendor.

Why the distribution is bimodal

The request has two parts, and the two parts have different costs, and the first part is the prefill, and the prefill is the reading of the prompt, and the reading is proportional to the prompt length, and the length varies, and the varying length is the first source of spread. The second part is the decode, and the decode is the generation of the output, and the generation is proportional to the output length, and the output length varies more than the input, because the model decides, and the deciding is the variance, and the variance is the spread.

The bimodality comes from the queue, because the provider's capacity is finite, and the finite capacity means the request either finds capacity or waits, and the finds capacity is the fast cluster, and the waits is the slow cluster, and the two outcomes are discrete, and the discrete outcomes are the two clusters, and the clusters are the shape, and the shape is the bimodal, and the bimodal is the reality, and the reality is the two numbers, and the two numbers are the two and the nine, and the average is the four, and the four is the lie.

The queue is the reason the distribution is not a smooth curve, because the queue is a threshold, and the threshold is the cliff, and the cliff is the gap, and the gap is the empty region, and the empty region is where the average lives, and the lives is the misleading, because the number is in the gap, and the gap has no requests.

Why the average is the wrong number

The average assumes a unimodal distribution, where the mean describes the typical request. A bimodal distribution has no typical request, so the mean describes nothing, because it sits in the empty gap.

The useless summary is worse than no summary, because the summary gets quoted, and the quoted number drives capacity planning, and the plan built on four seconds matches neither cluster. The fast requests beat the plan and the slow requests break it, and the broken ones are the user's experience, and the experience is the complaint the average was supposed to predict.

This is the same misreading as the TCP retransmission that became the p99, which is why the two fixes look alike.

What to measure instead

Report the percentiles, not the mean. The p50 is the typical, and the p95 is the bad day, and the p99 is the worst, and the three describe the distribution, and the distribution is the reality, and the reality is the user's experience, and the experience is the metric, and the metric is the fix, because the percentile describes an actual request and the mean does not.

Report the clusters separately. The fast cluster and the slow cluster are measured apart, and the apart is the shape, and the shape is the diagnosis, because the slow cluster's size is the queue's depth, and the depth is the capacity, and the capacity is the fix's target, and the target is the action, and the action is the scaling.

Measure time to first token and total separately. The first token is the prefill and the queue, and the total adds the decode, and the separate measurements show which part is slow, and the which is the diagnosis, and the diagnosis is the fix, because the prefill slow means the prompt is long and the decode slow means the output is long, and the two have different fixes.

Track the provider's queue, not only your latency. The provider's status page and the rate limit headers are the queue's signal, and the signal is the cause, and the cause is the distinction, because the slow cluster may be your traffic or the provider's load, and the distinction is the fix, and the fix depends on the cause.

The fixes

Scale the concurrency to the provider's capacity. The concurrency is the request rate, and the rate above the capacity is the queue, and the queue is the slow cluster, and the slow cluster is the latency, and the latency is fixed by the rate, and the rate is the semaphore, and the semaphore is the control, and the control is the fix.

Shrink the prompt. The prefill is proportional to the length, and the length is the prompt, and the prompt is the retrieval, and the retrieval is the context, and the context is the fix's target, per the context stuffing that inflated time to first token, and the common root is the property, not the platform.

Cap the output length. The decode is proportional to the output, and the output is the max tokens, and the max is the cap, and the cap is the bound, and the bound is the worst case's limit, and the limit is the tail's reduction, and the reduction is the p99's improvement.

Stream so the wait is visible. The streaming is the perceived latency, and the perceived is the experience, and the experience is the metric, and the metric is the fix, because the first token ends the wait, and the wait is the complaint, and the complaint is the incident.

The rule I keep

LLM latency is bimodal because the provider's queue is a threshold, so requests either find capacity or wait, and the mean of the two clusters lands in the empty gap between them. Report percentiles and the clusters separately, split time to first token from total, and fix the cause the measurement names.

The dashboard said four seconds and no request took four seconds, because the requests were two or nine and the average described the gap. I now check for that property first, because it is where the failure actually lives.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud