Requests, Limits and the QoS Class That Decided Who Died

The node ran out of memory and the kubelet evicted pods. The one that died was the database. The one that survived was a batch job. Nobody had written a rule saying "kill the database first". The rule was written anyway, at deploy time, by the requests and limits in two YAML files, because those numbers assign a QoS class and the QoS class is the eviction order.

This is the least read mechanism in Kubernetes. Everyone sets requests and limits to pass review, and almost nobody knows that the combination they chose places each pod in a priority ordering for death.

This was Kubernetes 1.32. The QoS rules are stable.

The three classes

Kubernetes computes a QoS class per pod from the relationship between requests and limits across all containers.

Guaranteed. Every container sets requests and limits, and they are equal, for both CPU and memory. This is the top tier. It says "this pod wants exactly this much and no more", and the kubelet treats it as the most worth keeping.

Burstable. Requests and limits are set but not equal, or only requests are set. This is the middle tier and the default shape of most deployments, because most people set a request lower than a limit to allow headroom.

BestEffort. No requests and no limits. The bottom tier. These go first.

The eviction order under memory pressure is BestEffort first, then Burstable, then Guaranteed. Within a class, the pod using the most memory above its request goes first. So the database, deployed as Burstable because someone wanted headroom, ranks below the system daemons that are Guaranteed, and above nothing in particular that mattered less than it did.

Why the database lost

The database was Burstable, request two gigabytes, limit eight, a sensible looking shape that lets it burst. The batch job was also Burstable but tiny. When the node came up short, the kubelet looked at Burstable pods and picked the one furthest above its request, which was the database, sitting at six gigabytes against a two gigabyte request.

The mechanism is doing exactly what it says: the request is the amount the scheduler promised and the kubelet defends, and usage far above the request is usage the pod borrowed. Borrowed memory is the first to be called in. The database was, in the kubelet's accounting, the largest debtor.

This is the same overcommit logic as noisy neighbour and oversold memory, but with an explicit eviction order instead of an implicit slowdown.

The traps in setting these numbers

Setting limits much higher than requests buys headroom and sells protection. The gap between request and limit is both your burst room and your eviction exposure. A wide gap means the pod can use a lot of memory that the node never planned for, and that is precisely the memory whose holder gets evicted. The numbers that look generous are the numbers that rank you low.

Omitting limits entirely makes you Burstable on requests alone, or BestEffort if you omit both, and BestEffort is a pod that volunteered to die first. "We do not set limits so it never gets throttled" is a sentence that trades CPU throttling for eviction priority.

The CPU and memory stories differ. Memory pressure evicts. CPU pressure throttles via the CFS quota, covered in container CPU throttling. A pod can be fine on CPU and still die on memory, because the two resources have different failure modes and different victims.

What right looks like

For the thing that must survive, the database, the cache that everything depends on, set requests equal to limits. That is Guaranteed, and it is a real statement: you are reserving exactly this much and giving up the right to burst, in exchange for being last in the eviction line.

For batch and disposable work, embrace the lower classes on purpose. BestEffort or low request Burstable is the correct declaration of "this is expendable", and it lets the kubelet protect the important pods by sacrificing the ones you meant to sacrifice.

For everything in between, keep the request honest, near the observed steady state, and the limit modestly above it. The request is what the scheduler uses to place you and what the kubelet uses to judge your borrowing, so a request set to a guess rather than a measurement misprices your pod in both markets.

Seeing the class you actually have

Do not infer it from intent. Read it:

kubectl get pod my-db -o jsonpath='{.status.qosClass}'

Do this for every pod that matters and you will find surprises, because the class is computed from the relationship of four numbers per container and small edits change it silently. A well meaning cleanup that removes a limit to fix throttling can demote a Guaranteed pod to Burstable and nobody notices until the next memory pressure event.

The rule

Requests and limits are not just scheduling numbers. They are a priority declaration that the kubelet enforces at the worst moment. Decide your QoS class on purpose: Guaranteed for what must live, low classes for what may die, and honest requests everywhere, because the gap between your request and your usage is the kubelet's list of debtors, and eviction is its collections process.

One more subtlety is worth naming: QoS is computed per pod, not per workload, so a database with three replicas can be Guaranteed on two nodes and Burstable on the third if a manifest variant slipped through, and the eviction order will then differ across your replicas for no reason anyone can see in the dashboard. Auditing the class across all replicas of the important workloads, not just one, is part of declaring it on purpose.

The observability side, knowing a node is about to run short before the kubelet decides for you, is the metrics story in three metrics that catch incidents.