Bug Hunt
The Slow Query That Ran in Four Milliseconds Under EXPLAIN
In production the query took nine hundred milliseconds. Under EXPLAIN it ran in four. The difference was the plan: production ran a cached generic plan…
Bug Hunt
One Cache Key, Two Customers, and a Very Bad Morning
Two accounts selling the same product slug saw each other's cached pricing. The key was built from the thing that was shared, and the thing that differed…
Bug Hunt
The Export That Always Returned Nine Hundred and Ninety Nine Rows
The finance team's export was short by a few rows and nobody knew by how many. The client library capped results at a thousand and raised no error, so the…
Bug Hunt
The Cron Job That Ran Thirteen Times in One Night
A nightly job was supposed to run once. The logs showed thirteen executions in a few hours, then none for a day.
Bug Hunt
File Uploads Corrupted Above One Megabyte and Nothing Below
Every file under a megabyte arrived intact. Every file over it arrived truncated at exactly the same boundary.
Bug Hunt
The Feature Flag I Flipped Three Hours Before Anyone Noticed
I turned the flag on at 9am. The first request that saw it arrived at noon, after a restart someone did for an unrelated reason.
Bug Hunt
Logged In Users Saw Somebody Else's Search Results
A support ticket said search was returning the wrong products. It was returning the right products for somebody else.
Networking
The WebSocket That Died Every Sixty Seconds and Nobody Owned Up
A live feed that dropped on a strict interval in production and never locally. The interval was the whole clue and I ignored it for a day.
JavaScript
The Dashboard That Got Slower the Longer You Left It Open
A single page app that was fine for an hour and unusable by the afternoon. Three wrong hypotheses, then a heap snapshot showed forty thousand detached DOM…
MicroVMs
The Service Is Running, Listening, and Completely Unreachable
curl works on the box and hangs from outside. Four places the packet can die, and how to tell which one killed it without guessing.
MicroVMs
The Test That Only Failed When It Ran Second
A suite that passed alone and failed in the pipeline. The bug was not in the test, it was in what the previous job left behind on the runner.
MicroVMs
I Rebooted the VM and the Kernel Did Not Change
A patched kernel that kept reporting the old version after every reboot. The reboot was working perfectly. My assumption about where the kernel comes from…