DevOps
Error Acquiring the State Lock, and the Four Things Not to Do
A stuck Terraform lock stops every deploy in the company, and the obvious fix, force unlock, is also the dangerous one.
Code Review
Reviewing Retry Logic Before It Becomes an Outage
A retry looks like resilience and behaves like a load generator. Every retry needs a budget, jitter and an answer to what happens if the first attempt…
Error Autopsy
Connection Reset By Peer at Exactly Sixty Seconds, Every Time
Every failure logged at the same number of seconds of idle time. That regularity is the diagnosis: a load balancer idle timeout killing connections your…
Security
The Webhook URL Field That Could Reach Your Metadata Endpoint
Any feature that fetches a URL the user supplies is a potential proxy into your internal network. The webhook settings page let an attacker reach the cloud…
DevOps
A Canary Deployment That Shipped a Broken Release Anyway
The canary looked healthy the entire time it was serving errors to real users. Five percent of traffic is not a sample, it is a rounding error, and your…
Tooling
perf record Turns a Mystery CPU Spike Into a Function Name
When CPU goes to 100 percent and your metrics show nothing, sampling the kernel's own profiler for thirty seconds will hand you the exact function responsible.
Deep Dive
How the Linux OOM Killer Chooses, and Why It Picked Your App
Under memory pressure the kernel kills one process and not another, and it is not random. The oom score is a small arithmetic formula you can read, tune and…
Bug Hunt
The Feature Flag I Flipped Three Hours Before Anyone Noticed
I turned the flag on at 9am. The first request that saw it arrived at noon, after a restart someone did for an unrelated reason.
Performance
Your Average Latency Is Fine and Your Users Are Not
Mean latency is a number that describes nobody. Here is why the tail dominates real user experience, how retries multiply it, and which percentiles are…
DevOps
The Thirty Seconds Between SIGTERM and SIGKILL
Kubernetes sends SIGTERM, waits, then sends SIGKILL. Most services lose requests in that window not because shutdown is hard but because nothing ever…
Opinion
Your Runbook Describes a System You Do Not Have
Runbooks get written when the system is designed and never tested again. The first incident shows you the gap, at the worst possible moment, to the person…
Networking
The WebSocket That Died Every Sixty Seconds and Nobody Owned Up
A live feed that dropped on a strict interval in production and never locally. The interval was the whole clue and I ignored it for a day.