Kubernetes Secret File Still Stale? Check subPath
A Secret update can reach Kubernetes without refreshing a subPath mount. Find where the rotation stopped and verify the application’s next read.
You rotate a Kubernetes Secret. The API reports the new revision, but the application keeps authenticating with the old credential. Before you tune kubelet refresh settings, look at the mount that supplies the file. Does it contain subPath?
A Kubernetes Secret mounted with subPath doesn't receive automatic updates in a running container. A normal Secret volume mounted as a directory can refresh eventually. The application still needs to reopen the updated path and reload the credential. Those are separate steps, and checking them in that order tells you where the rotation stopped.
This is an AI-prepared, documentation-based explainer, not a personally reproduced incident. The examples and staging checks are illustrative and were not executed. See Debugly's editorial policy.
The same Secret can reach the same pod two different ways
Imagine a service whose chart mounts one key at /etc/myapp/client-token. The chart author chose subPath so a volume wouldn't cover the other files already shipped in /etc/myapp. That's a sensible reason to want a single-file mount. It also gives that file a different update lifecycle from the Secret volume behind it.
# Illustrative fragment of a container's volumeMounts.
- name: client-credentials
mountPath: /etc/myapp/client-token
subPath: token
readOnly: true
The Kubernetes Secret docs distinguish two cases. For an ordinary Secret volume, changes propagate eventually; the delay depends on kubelet synchronization and its change-detection strategy. A subPath mount is the exception. Giving it another sync cycle won't make it follow the update. “Eventually” and “this mount doesn't update” are doing very different jobs here.
On Linux, the implementation makes that distinction less mysterious. The v1.36.5 AtomicWriter source shows how a volume switches its ..data symlink to a new generation. Opening a normal file path through the directory mount can then resolve to that generation. The Linux subPath implementation resolves the source and bind mounts it. That single-file mount doesn't follow the later symlink switch. Reopening the same mounted filename doesn't turn it into a fresh lookup through the Secret volume.
Find which of the three copies is stale
Check the live Pod, not only the values file that produced it. A chart template can add subPath without making the choice obvious in its configuration. This example prints container and mount metadata, not the Secret values:
# Illustrative read-only check; requires jq and permission to read the Pod.
kubectl -n your-namespace get pod your-pod -o json |
jq '.spec.containers[] | {name, volumeMounts:
[.volumeMounts[]? | {name, mountPath, subPath, subPathExpr}]}'
Match the volume name to the intended Secret reference in the Pod spec. Check the container that actually consumes the credential; a sidecar's fresh mount doesn't prove the app sees the same thing. If a deployment uses secretKeyRef instead, the process received an environment value at container start. Changing the Secret doesn't refresh that running process's environment.
| Where the evidence stops | What to check next |
|---|---|
| The intended Secret didn't change. | Check the writer, namespace and object name. Metadata can help correlate the update, but a new resource version alone doesn't prove the intended key changed. |
| The Secret changed; a fresh open of the file still sees the old version. | Check for subPath first. On a normal directory mount, allow for propagation and verify the actual Pod, mount and key mapping. |
| A fresh open sees the update; the app still uses the old credential. | Check whether it holds an open descriptor, parsed the file only at startup, or reuses already-authenticated connections. Reload and verify a new authentication. |
For a staging check, mount the same disposable Secret twice in a test container: once as a directory and once as a single file with subPath. Update a harmless marker from revision-a to revision-b. Reopen both paths after projection catches up. The expected distinction is a new marker in the directory mount and no automatic update at the single-file mount. Then check the application's reload behavior separately. Use dummy values for this experiment; don't print production credentials or put their hashes into logs.
Choose how you want updates to reach the process
If the application can read a different path, mount the Secret at a dedicated directory without subPath. You can still project just the one key using items:
# Illustrative fragments: volumeMounts on the container, volumes on the Pod.
volumeMounts:
- name: client-credentials
mountPath: /run/myapp-credentials
readOnly: true
volumes:
- name: client-credentials
secret:
secretName: client-credentials
items:
- key: token
path: client-token
Point the app at /run/myapp-credentials/client-token. A dedicated directory avoids hiding unrelated image files. The Kubernetes credential guide covers key projection and environment consumption. For reloads, have the app reopen the pathname and validate the replacement before swapping it into use. A file descriptor kept open across an atomic replacement still refers to what it originally opened; watching or polling should trigger a fresh open, not a reread of the same descriptor.
If the app requires the original single-file path or reads only at startup, keep the mount and make a controlled Pod replacement part of the rotation. Use the workload's normal rollout safeguards and verify fresh authentication before treating the rotation as complete. If the authority supports overlapping credentials, verify that the new credential is in use before revoking the old one. If it doesn't, plan for that constraint instead of assuming the rollout is seamless.
Debugly's ConfigMap restart story covers the startup environment case. Here the useful difference is that even a fresh file open can be stale: it's the mount that's pinned. Put the mount type and the application's reload contract side by side in the runbook. A green Secret update is the first check, not the last.