Reading /proc Like a Dashboard

The incident needed the process's thread count, its open file descriptors, its memory breakdown and its CPU time, and the monitoring agent, which was supposed to have all of it, had been restarted by the very incident and had a gap for the exact window in question. The machine, however, had been recording everything the whole time, in the one place that cannot crash, because it is the kernel: the /proc filesystem, which exports the kernel's own view of every process, as plain files, readable with the tools already on the box, which makes /proc the dashboard you never have to install, and the one that is always up when nothing else is.

I have run this failure in production myself, and the fix below is the one I now reach for first.

This is the reading guide for /proc as a live dashboard, the files that answer the incident questions, and the one line reads that produce the number.

This was a Linux 6.8 box, and every read below is a cat or an awk, which is the point: the dashboard is the kernel, and the client is a shell.

The process panel: /proc/PID

Each process is a directory named by its pid, and the files inside are its vitals, updated on read, which means they are always current, never stale, never sampled, which is a property no agent matches, because the agent samples and the kernel reports.

stat gives the state, the CPU time in user and system ticks, and the thread count, and a one line read of the fields answers "is it running, on CPU, and how threaded", which is the first panel of any process dashboard.

status is the human readable version, with the memory breakdown, VmRSS for the resident set, VmSwap for the swap, and the thread count again, and it is the file to read when the question is "how much memory, really", because RSS is the number that bites, per the OOM killer's arithmetic.

fd, the directory of open descriptors, counted, is the file handle panel, and its count against the limit from limits is the open files dashboard, which is the ulimit story from too many open files, readable live.

oom_score and oom_score_adj are the OOM panel, the kernel's current ranking of the process as a victim, which is the number that predicts the kill, and reading it during a memory incident is reading the future.

The system panel

Beyond the per process files, the system wide counters answer the fleet questions.

/proc/meminfo is the memory dashboard, MemAvailable being the number that matters, not MemFree, because available includes the reclaimable cache, and the gap between the two is the cache, which is the working set story.

/proc/loadavg is the run queue panel, the three averages being the demand for CPU, and the first number against the core count is the saturation ratio, above one is a queue, and a queue is latency, which is the CPU cousin of the connection pool queue.

/proc/diskstats and /proc/net/dev are the IO and network panels, counters that, read twice and differenced, give the rate, which is how every rate is really measured, as a delta over an interval, and the two read pattern is the dashboard's heartbeat.

/proc/pressure, where present, is the stall panel, the PSI counters that report the fraction of time tasks waited for CPU, memory or IO, which is the saturation metric the averages hide, and the closest the kernel comes to publishing its own p99 of waiting.

The reading discipline

The dashboard is not one read, it is a habit, and the habit has rules.

Rates are deltas: a single read of a counter is a number with no meaning, two reads an interval apart are a rate, and the interval is the resolution, so the one liner always samples twice, and the interval is stated, because a rate without its interval is a rumour.

The view is a snapshot, and the snapshot of a moving system is one draw, so the incident reading repeats, three samples, and the trend across them is the signal, which is the statistical discipline from the determinism post applied to the kernel.

And the read is cheap, so read often: the files are generated on access, with no storage cost, which means the /proc dashboard can be sampled at a rate an agent would never afford, and the high resolution is free, which is the property that makes it the incident tool, because the incident is exactly when you want resolution and the agent is exactly when you have gaps.

The one liners that earn their place

A few reads become muscle memory, and they are the dashboard's hotkeys: the thread count from status, the fd count from the fd directory, the RSS from status, the oom score, the load against cores, the memory available, and the pressure stalls, each a single awk or wc, and together they are the panel that answers, in five reads, the five questions every process incident begins with: running, threaded, leaking handles, eating memory, and waiting.

The agent has its place, the historical chart, the alert, the cross host view, and the /proc dashboard is not its replacement, it is its complement, the live, agentless, always up view for the moment the agent has a gap, which is the moment of the incident, which is the moment this post is for.

The rule

The kernel exports its view of every process and the whole system as plain files under /proc, current on read, crash proof, dependency free, so the dashboard for the incident is a handful of cats: state and threads and CPU from stat, memory and swap from status, handles from fd, the OOM ranking from oom_score, and the system saturation from loadavg, meminfo and pressure.

Read rates as deltas, snapshots as draws, and the incident as a trend, and the /proc dashboard is the one that is up when the agent is not, which is precisely when a dashboard is worth anything, and the kernel, which has never once needed a restart, is the most reliable collector you will ever read.