Three Metrics That Would Have Caught My Incidents
Not a guide to observability platforms. Just the three signals that, in retrospect, would have given me warning before nearly everything that went wrong.
| Resource | Saturation signal |
|---|---|
| CPU | Run queue length, or Linux pressure stall information |
| Memory | Page fault rate, swap activity, PSI memory |
| Disk | I/O queue depth, await from iostat |
| Connection pool | Requests waiting for a connection, and wait time |
| Thread pool | Queue depth |
| HTTP server | Accept queue backlog |
Related reading: The Memory Leak That Lived in the Metrics Pipeline explores a connected angle.
Related reading: Consumer Lag Is Not a Throughput Problem Until You Prove It Is explores a connected angle.
Related reading: Where Serverless Cold Start Time Actually Goes explores a connected angle.
Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud
