OOMKilled: Why Your Container Died and Your Metrics Showed Nothing

The short answer

Last State:  Terminated
  Reason:    OOMKilled
  Exit Code: 137

The kernel killed your container for exceeding its cgroup memory limit. Your dashboards showed nothing because they almost certainly graph the wrong number, either application heap or a sixty second average of a spike that lasted four seconds.

The number the kernel enforces is memory.current under cgroup v2, and it includes heap, thread stacks, code, page cache from file I/O, and kernel structures.

kubectl describe pod <pod> | grep -A5 "Last State"       # confirm OOMKilled
kubectl exec <pod> -- cat /sys/fs/cgroup/memory.stat      # the real breakdown

Tested on Kubernetes 1.32 with cgroup v2 on Linux 6.8.

Two different OOM kills

Worth separating immediately, because the fixes differ.

Container OOM, from the cgroup limit. Your container hit its own limits.memory. Only your container dies and Kubernetes restarts it. This is the common case and most of this post.

Node OOM, from system pressure. The whole node ran out of physical memory and the kernel's OOM killer picked a victim by badness score. Your pod can die even though it was well within its limit, because a neighbour over committed. Look for System OOM encountered in node events, or check dmesg on the node.

Tell them apart by whether the pod's own Reason is OOMKilled versus the node reporting system OOM events. If your pod is dying while comfortably under its limit, stop tuning your app and go look at the node.

Why your dashboard lied

This is where most of the time gets wasted. The graph says 400MB, the limit is 512MB, and the container was killed anyway.

You are graphing heap, not total memory

If your metric comes from inside the process, such as JVM heapUsed, Node's process.memoryUsage().heapUsed, or Python's tracemalloc, it measures only the language runtime's managed heap. The kernel counts everything.

A JVM with a 400MB heap routinely occupies over 700MB of RSS. A Node process doing image processing can hold hundreds of megabytes in native sharp allocations that heapUsed never sees.

Your scrape interval is longer than the spike

Prometheus at a thirty or sixty second scrape interval will simply miss a three second allocation spike. The kernel does not average. It kills the instant memory.current crosses the limit.

Any request handler that loads a whole file, builds a large JSON response, or decodes an image has a spiky profile. The average looks calm and the peak kills you. If your graph looks flat and your container dies, assume you are undersampling.

You are looking at RSS instead of working set

Kubernetes evicts and OOM kills based on working set, not RSS. Working set is roughly memory.current minus inactive file cache, meaning memory the kernel believes it cannot reclaim.

container_memory_working_set_bytes is the metric to alert on. container_memory_rss is not.

Page cache counts, and this surprises everyone

Under cgroup v2, pages your container reads from disk land in the page cache and are charged to your cgroup. A container that streams a 2GB file through, holding none of it in application memory, can drive memory.current to its limit.

The kernel will usually reclaim clean page cache under pressure rather than kill you. Usually. If allocation outpaces reclaim you get killed for memory that is, in a sense, not yours. This is why "my container OOMs during backups or log rotation or CSV export" is a real and confusing pattern.

Finding the real number

Ask the cgroup directly:

kubectl exec <pod> -- sh -c '
  cat /sys/fs/cgroup/memory.current
  cat /sys/fs/cgroup/memory.max
  cat /sys/fs/cgroup/memory.stat | head -20
'

memory.stat is the breakdown and usually contains the answer:

anon        312000000    # application memory
file        180000000    # page cache, often the surprise
kernel       12000000
slab         28000000
sock          4000000

memory.events gives you the history:

kubectl exec <pod> -- cat /sys/fs/cgroup/memory.events

low 0
high 4821
max 12
oom 3
oom_kill 1

max 12 means the limit was hit twelve times. oom_kill 1 means one actual kill. A high high count means the container is spending real time in reclaim, so it is thrashing before it dies and latency is already bad.

For peak rather than current, memory.peak exists on newer kernels. If it is unavailable, sample memory.current in a loop at one second resolution during a load test.

Runtime specific causes

JVM

The classic problem is a JVM that sizes itself against the node's memory rather than the container's. On a 64GB node with a 512MB container limit that goes badly. Modern JVMs are container aware, but set it explicitly anyway:

-XX:MaxRAMPercentage=70

Leave twenty five to thirty percent headroom for metaspace, code cache, GC structures, and thread stacks. Reserving ninety percent for heap is a reliable way to get OOMKilled with a heap graph that never goes above eighty five percent.

Thread stacks are a real cost at scale. Two hundred threads at 1MB each is 200MB that no heap metric shows.

Node.js

Node's default old space limit is around 4GB on 64 bit and is unrelated to your container limit. If your limit is 512MB, V8 will happily grow toward its own limit and get killed long before it decides to collect.

NODE_OPTIONS="--max-old-space-size=384"     # about 75% of a 512MB limit

Watch for Buffer allocations, which live outside the JS heap, and native modules like sharp, canvas, and better-sqlite3.

Python

Python's allocator often does not return freed memory to the OS. pymalloc manages arenas and an arena is only released when completely empty, so fragmentation from a workload allocating many small objects keeps RSS high even after the objects are gone.

Also, multiprocessing with fork shares memory copy on write initially, but a garbage collection pass touches refcounts on every object and dirties the pages, forcing a copy. A parent holding a large in memory dataset with eight workers can multiply memory use with no obvious allocation. gc.freeze() before forking helps.

Go

Go's garbage collector targets a heap ratio, so GOGC=100 means collect when the heap doubles, which is oblivious to any hard ceiling. Set a soft memory limit:

GOMEMLIMIT=400MiB

This makes the collector work harder as you approach the limit rather than sailing past it.

Requests, limits, and QoS

resources:
  requests: { memory: "256Mi" }
  limits:   { memory: "512Mi" }

requests is for scheduling, meaning how much the scheduler reserves. limits is for enforcement, meaning where the kernel kills you.

The gap between them sets your QoS class. When requests equals limits you get Guaranteed, which is last to be evicted under node pressure. When requests are lower than limits you get Burstable, evicted before Guaranteed pods. With neither set you get BestEffort, killed first.

For anything latency sensitive or stateful, set them equal. The apparent efficiency of a large gap is a bet that your neighbours will not burst at the same moment you do, and that bet loses at exactly the worst time.

A common anti pattern is setting no limit at all. That does not mean unlimited safety. It means your pod can trigger a node OOM and take down its neighbours, and the kernel's badness score will often pick the largest process, which is you.

When it is a genuine leak

If memory climbs monotonically across hours and never plateaus, you have a leak and no amount of limit raising will help.

Node. Take two heap snapshots twenty minutes apart under steady load and diff them in Chrome DevTools using the Comparison view. Sort by delta. The usual suspects are event listeners added per request, an unbounded Map used as a cache, and closures capturing large objects.

JVM. jcmd <pid> GC.heap_dump, then Eclipse MAT's leak suspects report. Add -XX:+HeapDumpOnOutOfMemoryError so you get a dump automatically, though note this fires on Java's own OOM and not on a cgroup kill.

That asymmetry is worth internalising. A cgroup OOM kill gives you nothing. No stack trace, no dump, no application log line. SIGKILL cannot be caught. If you need post mortem data you have to capture it before the kill, which means alerting at eighty five percent of the limit rather than investigating after.

Python. tracemalloc snapshots, or memray for a much better experience including native allocations.

Prevention

Alert at eighty five percent of the limit, not on the kill. Use container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.85 sustained for five minutes.

Graph working set, not heap. Put both on the same chart. The gap between them is your non heap usage and it is the thing nobody ever sees.

Set the runtime's internal limit to about seventy five percent of the container limit. JVM, Node, and Go all need to be told explicitly.

Load test with the production limit set. Most OOMs are discovered in production because staging had generous limits.

Scrape at ten seconds for memory sensitive services. A sixty second interval cannot see the spike that killed you.