Did Your Prometheus Sum Hide a Counter Reset?

One instance restarts, the others keep serving traffic, and the Prometheus total never goes down. That's exactly how a reset can disappear. If you sum the raw counters first, an increase on one instance can cover a drop on another. By the time rate() sees the total, the clue it needs has already been thrown away. For a fleet request rate, calculate rate() per counter series, then sum the rates.

This is an AI-prepared, documentation-based explainer, not a personally reproduced incident. The samples and queries below are illustrative and were not run against a live Prometheus server. See the editorial policy.

The total can look fine while one counter starts over

Imagine two instances of the same service. Between two scrapes, instance A restarts and its counter goes from 100 to 5. Instance B keeps running and goes from 100 to 210. Add the raw readings and the total goes from 200 to 215. A perfectly respectable little increase, if you ignore the restart that just happened underneath it.

On the original series, A's drop is visible, and B's increase is visible. On the sum, there's only a rise of 15. For these illustrative samples, the counter correction accounts for the 5 seen after A's reset plus B's increase of 110. That's an intuition for the missing signal, not a prediction of the exact rate() output: Prometheus extrapolates to the range boundaries, and requests between the last scrape and the restart may never have been observed.

The opposite shape can mislead you too. If the total drops, rate() can treat that aggregate drop as a reset of the whole series, even though only one instance restarted. The problem isn't that summing always undercounts. It's that the sum no longer represents one counter with one lifecycle.

Put the reset detection where the resets happen

The Prometheus function reference recommends applying rate() before aggregation so that it can see resets per series. Compare these illustrative queries for a conventional request counter:

# Per-series reset handling, then service aggregation
sum by (job) (rate(http_requests_total[5m]))

# Aggregate samples first; resets may be masked or misread
rate((sum by (job) (http_requests_total))[5m:])

The second expression uses a subquery, so it gives rate() a range of previously summed samples. A recording rule that stores the sum of raw counters loses the same information. No window setting can reconstruct which original series dropped from that sum. If you need a recording rule for the fleet rate, record the first expression instead. Choose the output labels intentionally: job is useful only if it defines the grouping you want.

Trace a suspicious dip back to an instance

1. Start with the underlying series. Graph the raw counter for a narrow job and time range around the anomaly. Retain the labels that separate the targets; if you collapse the labels here, you're back to the same problem.
2. Look for drops and missing data separately. resets(http_requests_total{job="api"}[5m]) counts decreases as resets for float samples. Compare with process restarts and scrape health. A decrease can be bad instrumentation rather than a real restart, and a replacement instance with a different label set may appear as a new series instead of a reset.
3. Compare the panel to the per-series rate. Evaluate rate() for the same targets and time window, then aggregate. If the panel still disagrees, check whether its recording rule already collapsed targets and whether the panel and rule use the same label scope, window, and evaluation timestamps.

Don't run this rule on a gauge. A gauge is allowed to go down, while a counter is expected to increase until it resets. Also, a rate needs enough samples: an instance with too little recent data may not contribute a result. Check that absence against scrape health rather than silently reading it as zero. For a different reason metrics can appear to disappear, see Debugly's OpenTelemetry cardinality overflow explainer.

Test the case where the total doesn't flinch

Give the recording rule a fixture with two instances: one resets while the other climbs enough to keep the raw sum increasing. Assert that the recorded fleet rate comes from the per-instance rates. Then add a case where an instance stops being scraped. That checks two different things: whether the rule handles a reset and whether the dashboard makes missing input visible. A fixture where every counter only rises won't tell you which query you wrote.