Prometheus Has Both Metrics. Why Does the Ratio Return No Data?

Both Prometheus inputs can contain data while their ratio is empty. Inspect matching labels, cardinality and missing denominators before changing the panel.

Share
Orange data cubes approach a matching gate with incompatible tag shapes, leaving the output tray empty.

Two queries draw perfectly ordinary lines. Divide one by the other and the panel goes blank. Apparently arithmetic has become an optional dashboard feature.

In Prometheus, a ratio between two instant vectors needs matching series before it can divide their values. If both inputs return float samples at the same evaluation time but the labels don't match, the result can be empty. Compare the full label sets, choose the labels that identify the same thing, and check that each match is unique before reaching for on, ignoring or group_left.

This is an AI-prepared, documentation-based explainer, not a personally reproduced incident. The samples, queries and diagnostic walkthrough below are illustrative and have not been executed against a Prometheus server. See the editorial policy.

The clue is in the labels, not the division sign

Imagine an exporter exposes storage readings using one gauge metric and a label to distinguish used space from capacity. At one evaluation time, the selected samples look like this:

storage_bytes{job="storage",instance="host-a",mount="/data",kind="used"}      40
storage_bytes{job="storage",instance="host-a",mount="/data",kind="capacity"} 100

The illustrative numbers use the same arbitrary unit. Their ratio should be 0.4. But this expression doesn't identify a pair:

storage_bytes{job="storage",kind="used"}
/
storage_bytes{job="storage",kind="capacity"}

The left series has kind="used"; the right has kind="capacity". Those values disagree. Selecting each value explicitly doesn't tell the binary operator to disregard the label afterward.

The Prometheus matching rules require a unique pair under the chosen matching labels. With default matching, the ordinary series labels must agree; different metric names alone don't prevent arithmetic matching. In this example, the meaningful join key is job, instance, mount. The kind label describes which measurement you're dividing.

storage_bytes{job="storage",kind="used"}
/ ignoring(kind)
storage_bytes{job="storage",kind="capacity"}

That is a one-to-one match if each remaining label combination has one used series and one capacity series. No group_left is needed. This is the same underlying mechanism behind a June 2018 community question about an empty storage ratio; it isn't evidence of a new Prometheus regression.

Prove which kind of “nothing” you've got

I'd open the expression browser's table view before changing a dashboard setting. Evaluate the two operands separately at the same timestamp, with the same filters and range window. A graph showing a line earlier in the range doesn't prove there's a usable sample at the instant you're inspecting.

Prometheus's selection and staleness rules explain why a stopped or stale series can disappear. A rate() input can also lack enough samples. Label matching can't manufacture missing measurements.

What you findWhat to investigate next
One operand is empty at the chosen instantFilters, scrape health, staleness and the rate window.
Both operands have float samples, but no matching label setsThe identity labels used by the ratio.
A matching modifier produces a uniqueness errorMultiple series now share the proposed join key.
A matched denominator is zeroUndefined or infinite arithmetic and how the panel renders it.

These are different failures. For ordinary float samples, division follows IEEE 754 arithmetic: zero divided by zero yields NaN, and a nonzero finite value divided by zero can yield infinity. An unmatched series instead contributes no result element. A panel may make those outcomes look similar; inspect the query result before deciding they mean the same thing.

This walkthrough assumes float samples. Native histogram arithmetic has additional restrictions; unsupported operand combinations can remove elements and produce annotations. Read the response's warnings or informational annotations if your data isn't ordinary gauge or counter-derived float data.

Choose the population before choosing a modifier

on(job, instance, mount) can express the storage match explicitly instead of saying “everything except kind.” Both approaches are useful, but they have different maintenance risks. ignoring(kind) still considers other labels added later. An explicit on list intentionally leaves them out. Neither choice knows whether leaving out a tenant, cluster or region would mix unrelated systems. That's your schema decision.

Requests make the question more interesting. Suppose you want one error fraction per job and HTTP method, across its instances. Aggregate both sides to that same population:

sum by (job, method) (
  rate(http_requests_total{job="api",code=~"5.."}[5m])
)
/
sum by (job, method) (
  rate(http_requests_total{job="api"}[5m])
)

Both operands now have the same intended label shape. The numerator counts the error subset; the denominator counts all requests in that population. Keep tenant or cluster labels on both sides if they're part of the population you need. Apply rate() before summing counters so resets remain visible per series, as explained in Debugly's counter-reset investigation.

If instead you need a separate fraction for each error code, the numerator retains code while the denominator remains one total per job and method:

sum by (job, method, code) (
  rate(http_requests_total{job="api",code=~"5.."}[5m])
)
/ on(job, method) group_left
sum by (job, method) (
  rate(http_requests_total{job="api"}[5m])
)

Here group_left declares the intended many-to-one relationship: several error-code series can use one denominator. It doesn't fix missing denominators or make a many-to-many relationship safe. The right side must still be unique for every matching key. Blindly adding it is rather like fixing a database join by asking for more rows and hoping nobody notices.

Check the join before trusting the percentage

For the storage example, this diagnostic counts denominator series under the proposed identity. Any returned count greater than one means the one-to-one assumption needs attention:

count by (job, instance, mount) (
  storage_bytes{job="storage",kind="capacity"}
) > 1

Then look for used series with no corresponding capacity series:

storage_bytes{job="storage",kind="used"}
unless on(job, instance, mount)
storage_bytes{job="storage",kind="capacity"}

These queries answer different questions: uniqueness and missing coverage. Run the uniqueness check on both operands for a one-to-one ratio, and on the designated one side for a many-to-one ratio. A blank count result alone doesn't prove the denominator exists.

Don't hide an empty join with or vector(0). That fallback can introduce an unlabeled zero; it doesn't restore the missing labeled relationships. Likewise, an exporter that emits an error series only after its first error needs an explicit absence policy. Missing telemetry is not automatically evidence of zero errors.

Before promoting the query to a recording rule or alert, give it fixtures for a valid pair, a missing denominator, a duplicate denominator and a zero denominator. For the request ratio, also include an absent error series and an instance reset. Choose the expected labels as well as the expected numbers. Prometheus provides rule unit testing with promtool; these are suggested checks, not tests performed for this article.

Put the intended join key and one-side uniqueness assumption next to the recording rule. The next label added to an exporter is then a reviewable schema change, rather than a surprise lesson in why two perfectly healthy lines can divide into nothing.

More on Debugly's observability articles. Prepared for Rohit Bhadani's Debugly publication; about Debugly.