> ## Content Index > Fetch the complete content index at: https://debugly.dev/llms.txt > Use this file to discover other available public pages before exploring further. # How a Single Lost Packet Becomes Your p99 - URL: https://debugly.dev/tcp-retransmit-becomes-p99/ - Published: 2026-10-09T13:19:00.000Z - Updated: 2026-10-23T09:12:51.000Z - Description: A dropped segment is not retried immediately. The retransmission timer backs off exponentially, so one loss adds a stall measured in round trips squared… - Author: Rohit Bhadani - Tags: Deep Dive, Networking, Performance The p99 chart showed a floor that no amount of compute moved, and the floor's height was suspiciously round, a second here, three seconds there, which are the heights of retransmission timeouts, not of slow code, because slow code scales with work while a retransmit stall scales with the loss and the backoff, and is paid in full by whichever request happens to lose a packet. The tail of a latency distribution is where the losses live, because a loss is rare, and the requests that experience one are rare, and rare plus expensive is precisely the definition of the tail. This is the mechanics of how one dropped segment becomes the p99, and why optimising the fast path never moves it, because the fast path is not where the loss lands. This was a service with a healthy median and a stubborn tail, on a path with a fraction of a percent of loss, and the arithmetic below is why the fraction, small as it is, owns the percentile. ## The retransmission timer, and its exponential patience TCP guarantees delivery by retransmitting segments that are not acknowledged, and the retransmission timer starts at roughly the measured round trip, but when the first retransmit is itself lost, the timer doubles, and doubles again, an exponential backoff, because the protocol assumes congestion and backs off to avoid adding to it. So the cost of a loss is not one round trip, it is the round trip plus the backoff sequence, and a single loss early in a connection, or a loss of the SYN itself, costs the initial timeout, commonly a full second or more, before the first retry even leaves. The stall is therefore quantised: a request that loses nothing is fast, a request that loses one segment pays the timeout plus the retransmit, and the distribution gains a second mode at the timeout height, which is the bump on the tail, and the bump's position is the timer, not the workload, which is why the p99 reads like a constant while the median breathes with load. ## Why the tail, specifically A loss is a per packet event with a small probability, and a request is many packets, so the probability that a request experiences at least one loss grows with the request's packet count, and the large responses and the long lived connections, the expensive requests, lose more often, which couples the loss to the already slow requests, deepening the tail twice. And because the loss is independent of the server's health, the tail persists on a perfectly healthy fleet, which is the confusing property: the p99 is bad while every server metric is green, because the loss lives in the path, not the fleet. This is the averages blindness from [p99 and why averages lie](https://debugly.dev/p99-latency-and-why-averages-lie/) with a concrete producer: the mean dilutes the rare second long stall across the fast majority, so the mean stays green while the percentile that matches the losing users burns. ## Where the loss lives The loss is rarely the server. It lives in the path: a congested queue on a transit link, a wireless hop, a saturated home router, a middlebox dropping under load, and the server observes it only as the retransmit counter, which is the path's confession in the server's own statistics. Reading the retransmit rate per connection, and correlating it with the tail, is the diagnosis: the requests with retransmits are the requests in the tail, and the correlation is near perfect, because the retransmit is the stall. The client side view complements it: the browser's or the mobile app's network log shows the stall at the segment level, and the HAR, per [reading a HAR file like a flight recorder](https://debugly.dev/har-file-flight-recorder/), shows the content download pausing for the timeout height, the pause being the retransmit made visible as a gap in the waterfall. ## The fixes that move the tail **Reduce the loss, because the stall is the loss.** The path's loss is the variable the stall multiplies, so the fixes that reduce loss move the tail directly: moving the server closer, using a better route, fixing the saturated link, and QUIC's recovery, which retransmits with faster, per packet timers and without the head of line coupling, per [HTTP/2 fixed head of line blocking then reintroduced it](https://debugly.dev/http2-head-of-line-blocking-vs-http3/), so a loss stalls one stream, not the connection. **Shrink the request's packet count.** Fewer packets means lower loss probability per request, so compression, smaller payloads and fewer round trips reduce the tail by reducing the exposure, which is the payload discipline from [Brotli saved forty percent on text](https://debugly.dev/brotli-vs-gzip-what-actually-helps/) applied to reliability rather than bytes. **Make the first byte early, so the loss lands after the response has started.** A loss before the first byte costs the whole timeout visibly; a loss mid stream is masked by the streaming and the retransmit overlaps useful progress, so moving TTFB earlier moves the loss into the masked region, which is a tail fix that does not touch the loss at all. **Watch the retransmit rate as a tail metric.** Graphing retransmits per request, and alerting on the rate, makes the path's loss a monitored quantity, so the tail's producer is visible before the percentile moves, which is the tripwire placed on the cause rather than the symptom. ## The rule I keep A dropped segment is retried after an exponentially backing off timer, so one loss costs a stall of round trips squared, and because loss is rare and per packet, the stall concentrates in the tail, where the expensive requests lose most and the mean never shows it. Reduce the loss, shrink the packet count, move the first byte early, and monitor the retransmit rate as the tail's producer. The p99 that reads like a constant is not slow code. It is the retransmission timer, a constant by design, paid by the unlucky, and the unlucky are a percentile, not a population, which is why the tail is the last place the fast path work ever reaches, and the first place the path work does. --- Related reading: [The Missing Intermediate Certificate That Broke Android](https://debugly.dev/missing-intermediate-certificate/) explores a connected angle. Related reading: [How Cloudflare Freed 100 TB by Deleting 64 Bytes at a Time](https://debugly.dev/cloudflare-100tb-dns-cache-bytes/) explores a connected angle. Published on [debugly.dev](https://debugly.dev/) by [Rohit Bhadani](https://debugly.dev/about/) | Founder of [Krova Cloud](https://krova.cloud/?ref=debugly.dev)