The ConfigMap Change That Nobody Restarted

Between the changed and the incident sit 6 quiet assumptions, and the incident is the story of one of them failing.

The configuration that is updated but not loaded has a single weak property, and here it is: the cluster state is correct and the process state is stale, so the manifest says one thing and the running process believes another, and both are true.

This was Kubernetes 1.30 with an environment-variable ConfigMap, and the mechanics are the same for any process that reads its configuration once at startup.

Why the pod kept the old value

The ConfigMap was the source, and the source was the environment variable, and the environment variable was the pod's start, and the start was the injection, and the injection was the once, and the once was the process's environment, and the environment was the immutable, and the immutable was the running process, and the process was the week, and the week was the old value.

The environment variable is set when the container starts, and the starts is the injection, and the injection is the snapshot, and the snapshot is the fixed, and the fixed is the process's life, and the life is the until restart, and the restart was the absent, and the absent was the week, and the week was the gap, and the gap is the reason the environment variable ConfigMap needs the rollout.

The mounted volume is different, and the different is the update, and the update is the file's change, and the change is the eventual, and the eventual is the kubelet's sync, and the sync is the minute, and the minute is the propagation, and the propagation is the file's content, and the content is the new value, and the new value is the still not loaded, because the process read the file at boot, and the boot was the once, and the once is the application's behaviour, and the behaviour is the fix's target.

Why the gap was invisible

The kubectl get showed the ConfigMap, and the showed was the new value, and the new value was the correct, and the correct was the assumption, and the assumption was the pod's behaviour, and the behaviour was the old, and the old was the invisible, because the manifest is the declared state and the process is the actual state, and the two diverged, and the diverged was the week.

The dashboard showed the deployment as healthy, and the healthy was the pods running, and the running was the true, and the true was the misleading, because the healthy does not mean the current, and the current is the configuration, and the configuration was the stale, and the stale was the incident, and the incident is the reason the configuration's version belongs in the observable state.

This is the same divergence as the rollback where the CDN kept serving the old version, and the common root is the property, not the platform.

The fix, in order

Restart the rollout after the ConfigMap change. The rollout restart is the pods' recreation, and the recreation is the new environment, and the new environment is the loaded value, and the loaded is the fix, and the fix is the command, and the command is the one line, and the line is the discipline, because the change without the restart is the change that did not happen.

Hash the config into the pod template. The annotation carries the ConfigMap's hash, and the hash is the template's change, and the change is the automatic rollout, and the automatic is the fix, and the fix is the tooling, and the tooling is the Helm or the Kustomize, and the two are the standard, and the standard is the prevention, because the config change becomes the deployment change.

Have the application watch the file. The mounted file is watched, and the watched is the reload, and the reload is the new value, and the new value is the fix, and the fix is the code, and the code is the SIGHUP or the inotify, and the two are the mechanism, and the mechanism is the application's job, because the platform updates the file and the application must notice.

Expose the loaded config version. The endpoint reports the configuration's hash, and the hash is the comparison, and the comparison is the gap's detection, and the detection is the fix, and the fix is the endpoint, and the endpoint is the observable, and the observable is the discipline, because the unverifiable configuration is the assumed configuration.

Alert on the divergence. The declared hash against the loaded hash is the check, and the check is the alert, and the alert is the detection, and the detection is the minutes rather than the week, and the minutes are the fix, and the fix is the monitoring.

What I now do

A pod reads its environment variables once at container start, so a ConfigMap change does not reach the running process until the pod restarts. Restart the rollout after the change, hash the config into the pod template so the change triggers a rollout, or have the application watch the mounted file, and expose the loaded config version.

The ConfigMap was updated and kubectl showed the new value and the pods kept behaving the old way for a week, because the environment variables were injected at container start and nothing restarted them. That is the whole pattern, and it is the part worth remembering.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud