Your process didn't crash. There's no stack trace, no exception, no core dump. One second it's running; the next, dmesg shows a single line — Out of memory: Killed process 4821 (node) — and it's gone. If you're running on Kubernetes, you'll see it as exit code 137 and a status of OOMKilled. No warning, no graceful shutdown hook, just a SIGKILL from the kernel itself.
This is the Linux Out-Of-Memory killer, and understanding how it decides who dies is the difference between a five-minute fix and a three-hour production incident.
Why Linux lets you overcommit memory in the first place
The OOM killer only exists because Linux's default memory allocation strategy is optimistic. When a process calls malloc(), the kernel usually hands back a virtual address range immediately without checking whether enough physical memory actually exists to back it. This is called overcommit, and it's controlled by vm.overcommit_memory:
sysctl vm.overcommit_memory
# 0 = heuristic overcommit (default)
# 1 = always overcommit, never refuse an allocation
# 2 = strict accounting, refuse once commit limit is hit
The reason this exists is fork(). Every time a process forks, the kernel logically duplicates its entire address space. A process using 4GB of RSS that forks would, under strict accounting, need to reserve another 4GB it will almost certainly never touch (thanks to copy-on-write pages). Refusing that fork because the memory 'isn't available' would break enormous amounts of software that relies on cheap forking. So the default heuristic mode (0) lets the allocation succeed and defers the actual reckoning until pages are touched, not when they're requested.
That deferral is exactly the gap where the OOM killer lives. Eventually, real physical pages get touched, swap fills up, and the kernel hits a point where it genuinely cannot satisfy a page fault. At that moment, something has to die, right now, synchronously, inside the kernel's page allocation path.
How the kernel picks the victim
The OOM killer doesn't kill the process that asked for the memory that pushed the system over the edge — it kills whichever process scores worst on a badness heuristic. You can read that score directly:
cat /proc/<pid>/oom_score
The calculation is roughly: take the process's resident memory (RSS) plus its swap usage as a percentage of total system memory, then apply an adjustment. That adjustment comes from oom_score_adj, a value from -1000 to 1000 that you or your process supervisor can set per-process:
cat /proc/<pid>/oom_score_adj # default: 0
echo 500 > /proc/<pid>/oom_score_adj # make it MORE likely to be killed
echo -500 > /proc/<pid>/oom_score_adj # make it LESS likely to be killed
echo -1000 > /proc/<pid>/oom_score_adj # exempt it entirely (sshd, systemd typically do this)
In practice this means: your biggest memory consumer usually dies first, but not always. A process with -1000 will survive even if it's using 90% of RAM, because the kernel treats it as unkillable. This is why sshd staying alive while your app server dies isn't luck — it's systemd-oomd or the kernel respecting a pre-set adjustment so you can still SSH in to investigate after an OOM event.
Containers change the blast radius
On a bare-metal box or a plain VPS, the OOM killer picks a victim from every process on the system. Inside a container orchestrated by Docker or Kubernetes, there's a second, more localized OOM killer at work: the cgroup memory controller.
When you set a container memory limit:
resources:
limits:
memory: "512Mi"
Kubernetes writes that as memory.max in the pod's cgroup v2 hierarchy. When the cgroup's memory usage — not the whole host's — exceeds that ceiling, the kernel invokes the OOM killer scoped to just that cgroup, and it picks a victim from processes inside it. This is a completely separate trigger from the system-wide OOM killer, and it's why a single pod can get OOMKilled while the node itself still shows plenty of free memory in free -h. Two different subsystems, two different thresholds, same underlying kill mechanism.
This distinction matters when you're debugging: dmesg -T | grep -i 'killed process' shows you host-level kills. For cgroup-level kills inside a container, you want:
journalctl -k | grep -i oom
cat /sys/fs/cgroup/<path>/memory.events # look for oom_kill counter
A rising oom_kill counter with a stable node-level memory graph is the signature of a cgroup limit that's too tight, not a real leak.
What to actually do about it
First, stop treating OOM kills as random. Check oom_score for your critical processes and make sure anything you can't afford to lose — your reverse proxy, your health-check sidecar, your database — has a negative oom_score_adj relative to batch jobs and workers, which should skew positive.
Second, if you're provisioning your own VPS rather than relying on a managed platform's defaults, decide deliberately between vm.overcommit_memory=0 (default, generally correct) and =2 (strict accounting, which trades OOM-kill risk for occasional allocation failures that your application has to handle explicitly). Strict mode is rarely worth it unless you're running a workload where a failed malloc() is safer than a killed process — think database engines that can gracefully reject a query versus a process that just dies mid-transaction.
Third, size container memory limits from observed peak RSS plus headroom, not from a guess. A limit set exactly at steady-state usage guarantees you'll get cgroup-OOM-killed the first time garbage collection, connection pooling, or a traffic spike pushes usage even slightly above baseline.
The OOM killer isn't a bug and it isn't arbitrary — it's a deterministic scoring algorithm doing exactly what it was configured to do. The failures that look mysterious are almost always a default oom_score_adj of zero on a process nobody thought to protect, or a cgroup limit copied from a template that never matched the workload's real memory profile. Once you can read /proc/<pid>/oom_score and cross-reference it against dmesg, the mystery disappears — it's just arithmetic.
