> ## Content Index > Fetch the complete content index at: https://debugly.dev/llms.txt > Use this file to discover other available public pages before exploring further. # The Autoscaler That Scaled on a Metric That Lagged the Traffic - URL: https://debugly.dev/hpa-lagging-metric-scale/ - Published: 2026-10-09T12:52:00.000Z - Updated: 2026-10-17T05:56:37.000Z - Description: The HPA added pods after the spike had passed and removed them after the next one began, because its metric was smoothed, delayed and averaged. - Author: Rohit Bhadani - Tags: DevOps, Kubernetes, Performance The traffic spiked, the latency spiked with it, and the autoscaler responded correctly and uselessly, by adding capacity four minutes after the spike ended, which then sat idle, and then removing it as the next spike arrived, which deepened it. The chart showed the replica count chasing the traffic curve with a fixed delay, oscillating around it, amplifying the wobble it was hired to damp, and the diagnosis was not a broken controller but a lagging signal, because a feedback loop is only as good as the freshness of its feedback. I have lost on-call weeks to this exact shape of slow burn, so the order below is the order I fix it in. This is the mechanics of the Horizontal Pod Autoscaler's loop, and of the three delays that turn it from a damper into an oscillator. This was Kubernetes 1.32 with an HPA on a custom metric, and the oscillation period matched the sum of the delays, which is the fingerprint that separates a tuning problem from a signal problem. ## The loop, and its three delays The HPA is a control loop: it reads a metric, compares it to a target, and adjusts replicas. Between the traffic changing and the replicas changing sit three delays, and each adds phase lag, and enough phase lag turns negative feedback into oscillation, which is control theory's oldest failure, in a manifest. **The metric delay.** The metric is scraped or aggregated on an interval, and a metric that is itself an average over a window, a five minute mean of CPU, is a value from the past by construction, reporting the spike after it has become history. The controller then acts on the average of a spike that is over, which is steering by a rear window. **The sync period.** The HPA evaluates on its own loop, default fifteen seconds, which is small, but the stabilisation windows, the downscale window especially, deliberately delay removals to avoid flapping, and the upscale decision is evaluated against the delayed metric, so the add arrives late by the metric's age plus the evaluation. **The pod readiness delay.** A new pod is not capacity until it is ready, and a pod that takes ninety seconds to boot, pulling a large image, running a slow startup, is capacity ninety seconds after the decision, which is the largest delay in most fleets, and the one the metric never includes, because the controller decides at decision time and the capacity arrives at readiness time. The sum of the three is the chase delay, and when the traffic's period is comparable to the chase delay, the loop oscillates, adding into the trough and removing into the peak, which is exactly the chart. ## Why the average is the wrong signal The metric's aggregation is the subtlest lag. CPU averaged over the fleet and over minutes hides the per pod saturation that users feel, so the controller sees a calm average while the p99 burns, and by the time the average rises, the tail has been burning for the averaging window. This is the [p99 and why averages lie](https://debugly.dev/p99-latency-and-why-averages-lie/) problem installed inside the control loop, where the loop optimises the average and the users live in the tail, and the loop is therefore blind to the very condition it exists to relieve. The probe coupling makes it worse: if the metric is derived from the same saturated path it protects, the metric degrades with the load, and the controller reads a failing signal at the moment it matters, the [readiness, liveness and the probe that restarted a healthy pod](https://debugly.dev/k8s-probe-restarted-healthy-pod/) coupling, one level up. ## The fixes that shorten the loop **Scale on a leading, fast metric.** Requests per second, queue depth, or in flight requests respond to traffic in seconds, not minutes, and lead the latency, so the controller acts while the spike is arriving, not after. Queue depth is the best leading signal because it is the integral of the imbalance, rising the moment demand exceeds capacity. **Report the tail, not just the mean.** Where the metric must be latency, use a high percentile of the metric, or scale on the parallelism the tail implies, so the loop sees the burn it exists to relieve. **Shrink the readiness delay.** The pod that boots in ten seconds is capacity ten seconds after the decision, and the boot time is dominated by the image pull, which the multistage slim image fixes, per [a one point two gigabyte image that should have been ninety megabytes](https://debugly.dev/dockerfile-multistage-slim-image/), and by prewarming, so the readiness delay stops being the largest term. **Tune the windows to the traffic's period.** The downscale stabilisation window must exceed the traffic's oscillation period, or the loop removes into every dip, and the upscale tolerance must admit fast adds, so the asymmetry, fast up, slow down, matches the asymmetry of the cost, where missing capacity is expensive and idle capacity is cheap. **Give the loop a floor.** A minimum replica count sized to absorb the known spike is the feedforward term, capacity placed before the signal, which is what every control system with a predictable disturbance uses, and the scheduled spike, the morning rush, the campaign launch, deserves scheduled capacity, not a surprised controller. ## The takeaway An autoscaler is a feedback loop, and its three delays, the metric's age, the evaluation period and the pod readiness time, add phase lag that turns damping into oscillation when the traffic's period matches the chase delay. Scale on a leading fast metric, report the tail, shrink readiness, tune the windows to the traffic's period, and place feedforward capacity for the predictable spikes. The replica count that chases the curve a fixed delay behind is not a broken controller. It is a correct controller with a stale signal, and the signal is the part of the loop everyone configures last, which is why it is the part that oscillates first. --- Published on [debugly.dev](https://debugly.dev/) by [Rohit Bhadani](https://debugly.dev/about/) | Founder of [Krova Cloud](https://krova.cloud/?ref=debugly.dev)