The latest Tag That Deployed Yesterday's Build

I have chased this from the incident end and from the succeeded end, and the truth was in the hand-off, as it usually is.

The mutable tag has a single weak property, and here it is: the tag is a name and not a version, so the same name can point at different images, and the node keeps the one it already has unless told to pull.

The case was a Kubernetes deployment pulling from a registry; the mechanics generalise to any container system with a mutable tag and a local cache.

Why the old image ran

The tag was the latest, and the latest was the pushed, and the pushed was the registry's new image, and the new image was the not the node's, and the node's was the cached, and the cached was the yesterday's, and the yesterday's was the running, and the running was the old code, and the old code was the incident.

The pull policy was the IfNotPresent, and the IfNotPresent was the cache's use, and the cache's use was the no pull, and the no pull was the old image, and the old image was the tag's ambiguity, and the ambiguity was the name against the digest, and the digest was the identity, and the identity was the not used, and the not used was the incident.

The deploy reported the success, and the success was the rollout, and the rollout was the new pods, and the new pods were the old image, and the old image was the cached, and the cached was the report's truth and the behaviour's lie, and the lie was the incident, and the incident is the reason the tag should be the immutable.

Why the report said the new version

The deployment's manifest said the latest, and the latest was the unchanged, and the unchanged was the no diff, and the no diff was the no rollout in some systems, and the rollout was the forced, and the forced was the new pods, and the new pods were the cached image, and the cached image was the old code, and the old code was the report's claim and the runtime's contradiction.

The version was the environment variable, and the variable was the build's stamp, and the stamp was the not updated, because the image was the old, and the old was the stamp's old, and the stamp's old was the clue, and the clue was the not read, and the not read was the incident's duration, and the duration is the reason the running version should be exposed and checked.

This is the same divergence as the ConfigMap change that nobody restarted, and the shared property is the one worth fixing.

The fix, in order

Tagged the image with the commit SHA. The tag became the immutable, and the immutable was the identity, and the identity was the new tag's new image, and the new image was the pull, and the pull was the fix, and the fix was the build's change, and the change was the discipline, because the mutable tag is the ambiguity.

Pinned the deployment to the digest. The digest was the content's hash, and the hash was the identity, and the identity was the unambiguous, and the unambiguous was the fix, and the fix was the reference, and the reference was the discipline, because the digest cannot point at two images.

Exposed the running version and asserted it after the deploy. The endpoint reported the SHA, and the SHA was the comparison, and the comparison was the deploy's verification, and the verification was the fix, and the fix was the check, and the check was the discipline, because the unverified deploy is the assumed deploy.

Set the pull policy explicitly for mutable tags. The Always was the pull, and the pull was the fresh, and the fresh was the fix, and the fix was the setting, and the setting was the mitigation, and the mitigation was the not the solution, because the Always is the slower start and the mutable tag is still the ambiguous.

Failed the deploy when the image was not found rather than falling back. The fallback was the silent, and the silent was the old image, and the old image was the incident, and the incident was the prevention's target, and the target was the loud failure, and the loud was the fix.

The rule I keep

A mutable image tag is a name rather than a version, so the same tag can resolve to different images on different nodes, and a cached node keeps the old one. Tag images with the commit SHA, pin deployments to the digest, expose the running version and assert it after every deploy, and fail loudly when the image is missing.

The deploy succeeded and the new code never ran, because the image tag was latest and the node already had an image with that name, so it used the cached one and the rollout reported a version the container did not contain. Get that property wrong and every layer above it lies to you politely.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud