How the Linux OOM Killer Chooses, and Why It Picked Your App

The incident report said the database "randomly crashed". It had not crashed. It had been killed, by the kernel, with a specific reason, at a specific moment, for a specific process. The kernel left a full explanation in the log. Nobody had read it.

The out of memory killer is not a mystery. It is a deterministic scoring function. Once you know the formula, "why did it kill my app and not the other thing" stops being a question and becomes arithmetic.

This was Ubuntu 24.04 with Linux 6.8, but the oom score has been stable in shape for a long time and the reasoning applies anywhere.

The kernel's actual problem

The kernel does not want to run out of memory. It wants to avoid a state where it cannot allocate a page for something essential. Before that happens, under genuine pressure, it picks one process to kill, frees its memory, and carries on.

The design goals, in order, are roughly: kill as few processes as possible, free as much memory as possible, and avoid killing something critical. The oom score is how it balances those.

The score

Every process has an oom_score, visible in /proc/<pid>/oom_score, from 0 to 1000. It is essentially the fraction of the system's memory the process is using, scaled, then adjusted.

Read it live:

ps -eo pid,comm --no-headers | while read pid comm; do
  echo "$(cat /proc/$pid/oom_score 2>/dev/null) $pid $comm"
done | sort -rn | head

The process with the highest score is the first candidate.

Underneath, the kernel computes oom_badness, which is: memory used by the process (RSS plus swap), plus memory held by its children that share memory, then adjusted by oom_score_adj, a value from -1000 to 1000 that userspace can set.

The adjustment is the interesting part:

A process with oom_score_adj = -1000 will never be killed. This is how critical daemons survive.

Why it picked your app and not the database

This is the question from the incident, and it usually has a concrete answer. Your application is often the biggest single consumer of memory. A Java heap, a cache, an in memory index. The database may be large too, but if it is smaller than the app, or if it has been given a protective oom_score_adj, the app loses.

Look at the OOM line in the log:

dmesg -T | grep -i "killed process"
# or
journalctl -k --since "yesterday" | grep -i oom

It prints the victim and, crucially, the full table of candidates with their scores at the moment of the kill. Reading that table is the single most informative debugging step in all of memory forensics, and almost nobody does it.

The log shows, for every process, its pid, name and score. The victim is simply the top. If you want the killer to have picked differently, the table tells you exactly which number to change.

The container subtlety

Here is where it gets genuinely surprising. In a container with a memory limit, the OOM event is usually not the system wide killer at all. It is the cgroup memory controller enforcing the limit, and the "victim" is often the container's whole process tree or a single process chosen by the cgroup.

So when your container is killed for exceeding its limit, that is not the oom killer making a choice. That is a hard limit doing exactly what it was configured to do, and the fix is the limit, not the score. This is the subject of OOMKilled and the container memory limit.

The system wide oom killer only gets involved when the host itself is under pressure, which in a well configured container deployment should be rare, because the cgroups are supposed to absorb the pressure first.

Tuning it deliberately

There are three levers, and they are all legitimate in the right place.

Protect the thing that must survive. Distributions already set sshd negative so you do not lose your shell mid incident. If you run a critical supervisor, give it a small negative adjustment:

echo -500 > /proc/$(pidof my-supervisor)/oom_score_adj

Sacrifice the thing that should die first. A cache process, a batch worker, anything easily restarted. Give it a high adjustment so the kernel takes it before it takes your database. Some batch frameworks do this for their workers on purpose.

Size the machine so the question never arises. The honest answer is that if the host is regularly invoking the oom killer, the real defect is overcommitment, and tuning scores is rearranging deck chairs. The killer is a last resort, and a system that uses its last resort weekly is misprovisioned.

Reading the kill, start to finish

When you get an OOM, do this in order.

First, find the kill in the kernel log and note the victim and the timestamp.

Second, read the candidate table printed with it. Confirm the victim had the top score, and see what was second. If the second was your database, you were one config line from a much worse incident.

Third, check the memory counters printed alongside, to see whether it was host pressure or a cgroup limit. The presence of cgroup memory events means the container limit fired.

Fourth, look at what the victim was doing. If it was your app and it had grown to several gigabytes, you have a leak or an unbounded cache, and the oom killer was the garbage collector you never wrote. That investigation is the heap snapshot work in Node memory leaks.

The mental model

The oom killer is not punishing you. It is an optimiser with a brutal objective function: free the most memory by killing the least valuable process, where value is approximated by a single number you can read and set.

Once you internalise that, the whole thing becomes legible. The victim was not unlucky. It won a scoring contest it was never told it was entered in.

If you find yourself reaching for oom_score_adj, also ask whether the process should be inside a cgroup with a real limit instead. The limit prevents the contest. The score only decides who loses it. Related reading on the noisy neighbour version of the same pressure: your database got slower and nothing changed.