The Memory Leak That Lived in the Metrics Pipeline

The Prometheus scraper was the thing being paged, not the application. Its memory climbed for weeks, restarts bought a day each, and the heap profile showed no leak in the classical sense, no growing allocation that was never freed. The growth was the time series index, and the series were real, created one per unique value of a label that someone had set to a user ID. The pipeline had not leaked. It had faithfully stored every identity it was handed, forever, which is the definition of unbounded cardinality.

This is the most common Prometheus incident, and it is worth its own autopsy because the defect is not in the tool. The tool did exactly what labels are for, and the defect is a modelling error that turns a metric into a database of individuals.

This was a Prometheus server scraping an exporter that attached a user id label, on a fleet of a few million users, and the arithmetic below is the whole story.

The symptom, precisely

Scraper memory and the series count grew together, linearly with the product's user growth rather than with its load, which is the fingerprint. A healthy service's series count is bounded by its code paths, endpoints, statuses, and stays flat as traffic grows. A cardinality leak's series count grows with the data, because the data is in the labels.

Restarts did not help for long, because the series are re created on the next scrape from the same labels, and the in memory index rebuilds to the same size. The only thing that shrinks it is removing the label, after which the old series age out of the retention.

The hypotheses that died

The scraper has a memory leak

First I profiled the scraper. The heap was dominated by the series index and the head blocks, which are the data, not a leak. Every byte was accounted for by stored series.

It died because the memory was honest. It held what it was given.

The application is emitting too much

Next I assumed the application emitted too many metrics. The metric count, the number of distinct metric names, was normal. The explosion was within one metric, across its label values.

It died because the defect was not volume of metrics but dimension of one metric, which is a different quantity and the one cardinality measures.

Retention will fix it

Then I hoped the retention window would prune it. Retention prunes time, not identity. A series that keeps receiving samples never ages out, and a user id label keeps receiving samples as long as users act, so the retention never sees the series go quiet.

It died because the leak was refreshed continuously by living traffic.

The breakthrough

The exporter's code showed the label:

requestsTotal.WithLabelValues(userID).Inc()

One line, and the metric's cardinality is now the user count, which is unbounded by definition. Every unique value of a label is a separate time series, stored, indexed, scraped and retained, and the cost is paid per series per scrape, forever. A few million users is a few million series, and the scraper's memory is the invoice.

The line was written to answer a question, which users are erroring, that the metric was never going to answer well, because a counter per user is a database of users, and the question belongs in the event log, not in the metric store.

Why cardinality is the whole game

Prometheus's model is a pull based store of series, where a series is a metric name plus a full label set, and every operation, scrape, index, query, retention, is per series. So the cost of a metric is the product of the cardinalities of its labels, and a label whose values are unbounded, user ids, request ids, urls with query strings, ips, multiplies the series count into the data's size rather than the code's size.

The model is brilliant for bounded dimensions, endpoint, method, status, region, because those are the questions operations asks, and it is catastrophic for unbounded ones, because those are the questions analytics asks, and the two question families need different stores.

The fixes

Remove the unbounded label. The immediate fix deletes the user id from the label set, after which the series count collapses to the bounded dimensions and the old series age out. The question the label was answering must move to the log or the event pipeline, where per event identity is cheap and queryable, which is the division of labour the incident violated.

Enforce cardinality at the exporter. A wrapper around metric creation that refuses, or hashes to a bounded bucket, any label whose value space is not declared, turns the modelling rule into a compile or runtime error instead of a quarterly incident. The declared value space is the contract, and the wrapper is the reviewer that never sleeps.

Alert on series count. The metric that predicts this incident is the scraper's series count and its growth rate, alerted on the slope, because a cardinality leak announces itself as a line with a gradient weeks before it announces itself as a page. The dashboard that watches the watcher is the tripwire.

Budget cardinality in review. The review question for any new metric is the product of its label cardinalities, stated as a number, and a number over a few thousand is a blocking comment, because the cost is paid per scrape forever, which is the same forever arithmetic as the index tax in the index that made inserts four times slower.

What I would do differently

I would have treated a label as a schema decision with a value space, because that is what it is, and the value space is the capacity plan. The line that added the user id was a schema change to a database that happened to be written in monitoring vocabulary, and it shipped without the review a schema change gets.

The rule I keep

A Prometheus label's value space is the metric's cardinality, and an unbounded value space is an unbounded database growing with your data, paid for per scrape forever. Keep labels to bounded dimensions, move per identity questions to the event log, enforce the value space at the exporter, and alert on the series count's slope.

The leak was never in the code that allocates. It was in the one line that chose a dimension, which is the lesson of every key built from too little, from the cache that returned another customer's data to this one.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud