Error Autopsy
The Connection String That Broke on a Special Character
The rotation generated a strong password with an at sign in it, and the connection string, parsed as a URL, read the at sign as the user host separator, and…
DevOps
The Failover That Promoted a Replica Hours Behind
The primary died and the replica took over in seconds, which is what the dashboard promised. The last three hours of writes were not there, because the…
Performance
Your RAG Is Slow Because Retrieval Is an Untuned Database
The LLM answered in four hundred milliseconds and the request took four seconds. The gap was retrieval, a vector search over an index nobody sized, chunked…
Performance
The Query That Was Fast for Two Years and Then Was Not
Nothing about the query changed. The table crossed the size where the planner stops choosing the index and starts scanning, because the working set stopped…
DevOps
A Backup You Have Never Restored Is a Rumour
The backup completed every night and the restore failed in the first hour of the drill. A backup is a claim about the future restore, and claims that are…
Code Review
Reviewing Pagination Before Someone Requests a Million Rows
Offset pagination that scans, an unbounded page size, and a cursor that drifts when rows insert. The pagination diff looks like plumbing and contains the…
Bug Hunt
Everything Broke the Day We Doubled the Instance Count
Scaling out doubled the instances and, invisibly, doubled the connection pool, past the database's limit. The database was the bottleneck the day the team…
Deep Dive
Vacuum, Dead Tuples and the Wraparound Shutdown Nobody Plans For
MVCC gives every transaction an id and keeps old row versions until vacuum says they are dead to everyone. If vacuum falls behind for long enough, the id…
Error Autopsy
relation does not exist for a Table That Is Definitely There
The query fails with relation does not exist while the table sits in the schema browser a click away. The table never moved.
Performance
Two Hundred Redis Calls, and Every One Was a Round Trip
Each Redis command took a tenth of a millisecond to execute and a full round trip to deliver, and two hundred of them in a loop made the request slow while…
Code Review
Reviewing a Cache Layer for the Four Ways It Will Hurt You
A cache is reviewed for correctness on the happy path and fails on the edges: the cold start, the hot key, the poison entry and the deploy.
Bug Hunt
The Slow Query That Ran in Four Milliseconds Under EXPLAIN
In production the query took nine hundred milliseconds. Under EXPLAIN it ran in four. The difference was the plan: production ran a cached generic plan…