The Helm Value You Set and the One That Actually Applied

Share
The Helm Value You Set and the One That Actually Applied. Abstract devops illustration in orange and dark grey on debugly.dev

The replica count in the values file said three. The deployment in the cluster said one. Both were true. The one that applied came from a place nobody had edited in months, which is exactly why nobody could find it.

Helm's value precedence is a short, fixed list, and the bugs it produces are all the same shape: a value set in a higher precedence source quietly beats the one you are looking at. The fix is not more editing. It is knowing the order and making the effective values visible.

This was Helm 3.x against Kubernetes 1.32. The precedence rules are stable across versions.

The precedence order

From lowest to highest, later wins:

  1. The chart's own values.yaml.
  2. A parent chart's values for a subchart, under the subchart's key.
  3. Values files passed with -f, applied in order, later file wins.
  4. Individual --set arguments, which beat all files.

Within --set, there is a further ordering by type, with --set-string, --set-file and --set-json each handling their input, but all of them beating any file.

That is the whole list, and it is the entire mental model. The value you are staring at in values.yaml loses to anything above it, and the most common source of surprise is a -f file or a --set in a CI command that somebody added long ago and forgot.

Seeing the effective values

The command that ends every argument is rendering without deploying:

helm template myrelease ./chart -f prod.yaml --set replicas=5 | grep -A2 replicas

helm template shows the exact manifests that would be applied, with all precedence resolved. If the rendered manifest says one and your file says three, the difference is a higher precedence source, and the template output is the ground truth to diff against your expectation.

For a release already deployed, helm get values shows what was recorded:

helm get values myrelease --all

--all includes the computed effective values, merging the chart defaults with the user supplied ones. Comparing helm get values against your repository is how you find the drift where CI is passing a --set that is not in the repo.

The traps that precedence creates

The forgotten CI --set. A one off --set image.tag=hotfix added during an incident becomes permanent when it is left in the pipeline. Every later deploy reapplies it over the top of your carefully managed values file. The rendered manifest shows it. The repo does not. This is the "value applied from a fourth place" incident.

Multiple -f files with overlapping keys. Teams accumulate base.yaml, prod.yaml, prod-eu.yaml. The order on the command line decides the winner, and a reordered command changes behaviour with no diff. The order is part of the configuration and should be reviewed like code.

Subchart values living in the parent. To configure a subchart you set values under its name in the parent's values, and those beat the subchart's own defaults but lose to a --set with the full path. It is easy to edit the subchart's values file and wonder why nothing changes, when the parent was overriding it all along.

Type surprises with --set. --set replicas=3 yields a number, but --set name=03 becomes a number and drops the leading zero, and commas split into lists. Values that must stay strings belong in --set-string. A silently coerced type is a value you set and a different value that applied.

Making it boring

A few habits remove this whole class of surprise.

Render in CI and diff the manifests against the previous release, so a value change anywhere in the precedence chain shows up as a reviewable diff before it touches the cluster. This turns precedence from a mystery into a code review.

Keep the source of truth in one place. If prod values live in a file in the repo, do not also pass them as --set from the pipeline. Every additional source is another layer that can beat the one you read.

Name your values files in the order they are applied, and put the apply command itself in the repo, so the precedence order is visible and versioned rather than living in a CI UI.

A schema as the precedence contract

One structural fix reduces this whole class of confusion: ship a values.schema.json with the chart. The schema documents every value the chart consumes, its type and its default, and it rejects unknown or mistyped values at install time.

The schema matters for precedence bugs specifically because so many of them are typos. A value set under the wrong key does not error in Helm. It is simply ignored, and the chart falls back to a lower precedence default, which is exactly the "I set it and it did not apply" complaint. With a schema, the unknown key fails loudly at template time instead of silently at runtime, converting the worst kind of precedence bug, the silent one, into a build error.

It also makes the effective surface reviewable. When the schema is the contract, a --set that is not in the schema is visible as an exception to the contract, and the fourth place that set your value loses its ability to hide.

The rule

Helm resolves values by a fixed ladder, and the value you read is only the value you get if nothing higher on the ladder disagrees. Render with helm template before you deploy, read helm get values --all after, and treat any difference between your editor and the rendered manifest as a higher precedence source waiting to be found.

The deployed truth versus the repository truth gap is the same defect class as the runbook that describes a system you do not have, in your runbook describes a system you do not have, except here the machine will show you the truth if you ask it to render.