Code Review
Every Timeout Should Be Shorter Than the One Above It
A request chain is a ladder of deadlines, and when a lower rung is longer than the one above, the upper layer pays for work nobody will ever read.
Opinion
The People Who Designed It Should Be on the Pager
Escalation paths that route incidents away from senior engineers also route the feedback away from the people who can fix the design.
Deep Dive
Your Container Has Two CPUs and Is Using One Point Three
The container's CPU limit is two cores and its usage never exceeds one point three, yet requests slow down. CFS quota throttles in hundred millisecond…
DevOps
Requests, Limits and the QoS Class That Decided Who Died
When a node runs short of memory, Kubernetes does not choose eviction victims at random. It uses the QoS class, which your requests and limits silently…
Tooling
journalctl Is a Database and You Are Using It Like a Text File
Grepping a wall of journal output loses the structure the journal already keeps. It is a queryable database of structured fields, and the queries take…
Opinion
Nobody Looks at Your Dashboard Until Something Is Already Broken
Dashboards are built for demos and read during incidents, which is the one moment they cannot help. Monitoring that is only useful after the fact is…
Error Autopsy
504 Gateway Timeout While Your Application Was Still Working
The proxy gave up at sixty seconds and returned a 504. The application finished at seventy four and wrote the result to nobody. Both were correct.
DevOps
The Helm Value You Set and the One That Actually Applied
You set the value in three places and the cluster applied a fourth. Helm's precedence rules are short and unforgiving, and the gap between what you edited…
Tooling
lsof and ss Will Tell You Who Owns That Port
Port in use, connection stuck, file handle exhausted. Three of the most common production questions are answered by two old tools that read the kernel's own…
Opinion
Most of Your Microservices Should Be Modules
A network call is a thousand times slower than a function call and a hundred times harder to debug. Most service boundaries were drawn for organisational…
Bug Hunt
The Cron Job That Ran Thirteen Times in One Night
A nightly job was supposed to run once. The logs showed thirteen executions in a few hours, then none for a day.
DevOps
Error Acquiring the State Lock, and the Four Things Not to Do
A stuck Terraform lock stops every deploy in the company, and the obvious fix, force unlock, is also the dangerous one.