> ## Content Index > Fetch the complete content index at: https://debugly.dev/llms.txt > Use this file to discover other available public pages before exploring further. # The Node That Evicted Everything Because Logs Filled the Disk - URL: https://debugly.dev/node-disk-pressure-evicted-pods/ - Published: 2026-10-09T15:19:00.000Z - Updated: 2026-10-17T05:56:07.000Z - Description: Container logs grew without rotation until the node hit disk pressure, and the kubelet evicted every pod on it, including the healthy ones, and the eviction… - Author: Rohit Bhadani - Tags: DevOps, Kubernetes, Observability Start at the restarted and you miss it. Start at the eviction and you miss how it arrived. The bug lived in the hand-off. Here is the shape of the disk pressure eviction, and the one property that makes it look like an application failure: the pods are killed by the node rather than by their own crash, so the restart count rises and the logs show no error, because the error is the node's disk, not the pod's code. This was Kubernetes 1.30 on three nodes, and the mechanics are the same for any cluster where container logs are not rotated. ## Why the disk filled The container wrote to stdout, and the stdout was the log, and the log was the file, and the file was the node's disk, and the disk was the finite, and the finite was the growth, and the growth was the unrotated, and the unrotated was the configuration, and the configuration was the default, and the default was the no limit, and the no limit was the months, and the months was the full, and the full was the pressure. The rotation was the container runtime's job, and the runtime's rotation was the setting, and the setting was the size and the count, and the size was the file's maximum, and the count was the files kept, and the two were the bound, and the bound was the absent, and the absent was the growth, and the growth was the incident, and the incident is the reason the runtime's log settings belong in the node's provisioning. The growth was slow, and the slow was the invisible, and the invisible was the no alert, and the no alert was the disk metric, and the metric was the absent, and the absent was the surprise, and the surprise was the eviction, and the eviction was the incident, and the incident is the reason the disk usage belongs on the dashboard with a threshold. ## Why the eviction looked like a crash The kubelet evicted the pods, and the evicted was the kill, and the kill was the restart, and the restart was the count, and the count was the CrashLoopBackOff, and the BackOff was the appearance, and the appearance was the application, and the application was the suspect, and the suspect was the wrong, because the application was fine, and the fine was the node, and the node was the disk, and the disk was the eviction's cause, and the cause was the invisible in the pod's logs, because the pod did not crash, it was killed. The distinction is in the event, and the event is the node's, and the node's event says evicted, and the evicted is the disk pressure, and the pressure is the diagnosis, and the diagnosis is the kubectl describe, and the describe is the reading, and the reading is the fix's entry, and the entry is the event rather than the log, and the event is the node's voice, and the voice is the truth. This is the same misdirection as [the Kubernetes probe that restarted a healthy pod](https://debugly.dev/k8s-probe-restarted-healthy-pod/), and the common root is the property, not the platform. ## The fixes **Set the container runtime's log rotation.** The runtime caps the file size and the file count, and the cap is the bound, and the bound is the disk's protection, and the protection is the fix, and the fix is the node's configuration, and the configuration is the provisioning, and the provisioning is the discipline, because the unrotated log grows without limit and the limit is the setting. **Alert on the node's disk usage.** The metric is the available space, and the threshold is the eighty percent, and the eighty is the warning, and the warning is the time, and the time is the fix before the eviction, and the eviction is the incident, and the incident is the prevention's target, and the target is the alert. **Set ephemeral storage limits on the pods.** The limit is the pod's bound, and the bound is the eviction's trigger at the pod level, and the pod level is the contained, and the contained is the fix, because the one pod that writes too much is evicted rather than taking the node, and the node is the blast radius, and the radius is the reduction. **Ship the logs off the node.** The logs are collected and the local files are rotated aggressively, and the aggressive is the small local footprint, and the footprint is the disk's usage, and the usage is the bound, and the bound is the fix, and the fix is the collector, and the collector is the pipeline, and the pipeline is the discipline, because the node's disk is not the log store. **Reserve space for the kubelet.** The node's allocatable excludes the system reserve, and the reserve is the kubelet's room, and the room is the eviction's avoidance, and the avoidance is the fix, and the fix is the configuration, and the configuration is the scheduling, because the pods should not be able to fill the disk the kubelet needs. ## What I now do Container logs written to the node's disk grow without bound unless the runtime rotates them, and a full disk makes the kubelet evict every pod on the node, including the healthy ones. Set log rotation at the runtime, alert on node disk usage, set ephemeral storage limits, ship logs off the node, and reserve space for the kubelet. The pods looked like they were crashing and the application was fine, because the node's disk had filled with unrotated container logs and the kubelet evicted everything running on it. I now check for that property first, because it is where the failure actually lives. --- Published on [debugly.dev](https://debugly.dev/) by [Rohit Bhadani](https://debugly.dev/about/) | Founder of [Krova Cloud](https://krova.cloud/?ref=debugly.dev)