The Volume That Would Not Grow Until the Pod Restarted

I would like to say the missing step was the problem, because that would be a shorter post, but it started at the full.

The volume expansion that stops at the block device has a single weak property, and here it is: the claim reports the new size, so the platform's view is correct, and the filesystem inside the pod still reports the old one, and the two are true at the same time.

This was Kubernetes 1.30 with a CSI volume, and the mechanics are the same for any storage system where the block device and the filesystem are resized in separate steps.

Why the filesystem did not grow

The claim was the request, and the request was the cloud's expansion, and the expansion was the block device, and the block device was the twenty, and the twenty was the raw capacity, and the raw capacity was the not the filesystem, and the not filesystem was the growpart and the resize2fs, and the two were the second step, and the second step was the not run, and the not run was the ten, and the ten was the application's limit.

The online expansion is the capability, and the capability is the storage class's, and the class's is the allowVolumeExpansion, and the flag was the true, and the true was the block's growth, and the growth was the filesystem's requirement, and the requirement was the kubelet's job, and the job was the conditional, and the conditional was the pod's state, and the state was the not always triggering, and the not triggering was the restart.

The two sizes are the different layers, and the layers are the block and the filesystem, and the filesystem is the application's view, and the view is the ten, and the ten is the error, and the error is the incident, and the incident is the reason the expansion should be verified from inside the pod rather than from the claim.

Why the dashboard said it was fine

The dashboard read the claim, and the claim was the twenty, and the twenty was the platform's truth, and the truth was the not the application's, and the not application's was the gap, and the gap was the invisible, and the invisible was the incident's duration, and the duration was the hour, and the hour was the verification's absence, and the absence is the reason the expansion needs the inside check.

The metric was the node's disk, and the node's disk was the block's size, and the block's size was the twenty, and the twenty was the misleading, and the misleading was the fine, and the fine was the contradiction, and the contradiction was the diagnosis' entry, and the entry was the df inside the pod, and the df was the ten, and the ten was the answer.

This is the same divergence as the ConfigMap change that nobody restarted, same mechanism, different costume.

The fix, in order

Verified the filesystem size from inside the pod after the expansion. The df was the check, and the check was the filesystem's actual size, and the size was the verification, and the verification was the fix, and the fix was the step, and the step was the discipline, because the claim's size is the not the application's.

Restarted the pod where the online resize did not trigger. The restart was the kubelet's re-run, and the re-run was the resize2fs, and the resize2fs was the filesystem's growth, and the growth was the fix, and the fix was the rollout restart, and the restart was the discipline, because the expansion without the filesystem growth is the no-op.

Set allowVolumeExpansion and checked the storage class supports the online resize. The flag was the permission, and the permission was the expansion, and the expansion was the capability, and the capability was the fix, and the fix was the class, and the class was the discipline, because the class without the flag rejects the request.

Alerted on the filesystem's usage, not the claim's size. The metric was the df's percentage, and the percentage was the application's reality, and the reality was the alert, and the alert was the prevention, and the prevention was the fix, and the fix was the exporter, and the exporter was the discipline.

Documented the expansion as the two steps. The steps were the claim and the filesystem, and the two were the runbook, and the runbook was the verification, and the verification was the fix, and the fix was the document, and the document was the discipline, because the single-step assumption is the incident.

The rule

Expanding a persistent volume claim grows the block device, and the filesystem inside it grows only when the resize step runs, which may require a pod restart. Verify the filesystem size from inside the pod after every expansion, alert on filesystem usage rather than claim size, and confirm the storage class supports online expansion.

The claim showed twenty gigabytes and the application kept failing with no space left, because the filesystem inside the pod was still ten and nothing had run the resize step. I now check for that property first, because it is where the failure actually lives.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud