Bug Hunt
The Memory Leak That Lived in the Metrics Pipeline
The scraper's memory grew without bound and no application leaked. Someone put a user ID in a Prometheus label, and every unique value became a permanent…
Performance
Event Loop Lag Is the Metric Nobody Alerts On
CPU at forty percent, memory fine, and requests timing out. The Node process is not busy, it is blocked, and the only metric that shows the difference is…
DevOps
A Backup You Have Never Restored Is a Rumour
The backup completed every night and the restore failed in the first hour of the drill. A backup is a claim about the future restore, and claims that are…
Bug Hunt
Everything Broke the Day We Doubled the Instance Count
Scaling out doubled the instances and, invisibly, doubled the connection pool, past the database's limit. The database was the bottleneck the day the team…
DevOps
The Ingress Annotation That Silently Changed Your Timeout
The app's timeouts were tested and correct, and the 504s kept coming, because the ingress controller has its own defaults for body size, buffering and read…
Tooling
Ten bpftrace One Liners That Replace a Week of Guessing
Latency without CPU, stalls without errors, syscalls nobody counted. The kernel already observes all of it, and bpftrace turns the observation into a one…
DevOps
Six Months of Running MicroVMs in Production
What broke, what surprised me, and what I would tell someone considering Firecracker for their own platform.
DevOps
A 1.2 GB Image That Should Have Been 90 MB
The image shipped the compiler, the package cache and the build tools to production, because every layer ships.
Error Autopsy
fork: Resource Temporarily Unavailable With 8 GB Free
The process could not fork despite plenty of free memory, and the error says resource temporarily unavailable. The resource is not RAM.
Security
Deleting the Commit Did Not Delete the Secret
The credential was removed from the current tree and the team considered it handled. The blob still lives in history, in every clone and in every fork, and…
DevOps
Your Node Drain Hung Because of a Pod Disruption Budget
The node upgrade stopped at one node and the drain sat for hours. A Pod Disruption Budget with minAvailable equal to the replica count means nothing can…
Tooling
git bisect run Finds the Commit While You Get Coffee
Manual bisection is careful, slow and error prone. Automated bisection with a script that returns the right exit codes finds the offending commit in log…