DevOps
The Prompt and the Model That Could Not Be Separated
The rollback reverted both changes at once, so the fix for the prompt also undid the model upgrade, and the incident lasted until someone split the two and…
Error Autopsy
The Token Budget Kill Switch That Never Fired
The guard was armed, the alert was wired, and the switch never fired, because it compared a token count against a dollar limit, and the units never matched…
DevOps
The Model Upgrade That Changed the Output Schema Nobody Pinned
The model version floated, the provider shipped an update, and the JSON the parser expected gained a wrapper, so every request parsed to empty and nothing…
DevOps
The Secret Scanner That Ran After the Push
The check that detects the leaked credential ran in CI, which runs after the push, which is after the credential is cloned by every mirror, fork and CI runner.
Code Review
Reviewing a Terraform Diff for the Resource It Will Replace
A one character change to an immutable attribute shows as a tidy edit in the diff and executes as a destroy plus create, and the review that prices the edit…
WooCommerce
The WooCommerce Cron That Never Ran Without Visitors
The scheduled tasks were configured and due, and none ran, because the scheduler only fires on a page load, and the store was quiet, and the quiet period…
Security
The Egress Rule That Let the Backup Phone Home
The backup job needed one destination, the egress allowlist gained a wildcard to make a Tuesday work, and the exfiltration path stood open for a month…
DevOps
The Autoscaler That Scaled on a Metric That Lagged the Traffic
The HPA added pods after the spike had passed and removed them after the next one began, because its metric was smoothed, delayed and averaged.
DevOps
The Word Doing All the Work in the AWS Data Loss Notice
AWS confirmed in September that data held exclusively in me-south-1 and mec1-az2 cannot be restored. Multi-AZ worked as designed.
DevOps
The Failover That Promoted a Replica Hours Behind
The primary died and the replica took over in seconds, which is what the dashboard promised. The last three hours of writes were not there, because the…
DevOps
The Liveness Probe That Killed the App When the Database Was Slow
The probe queried the database, so the slow database made the app look dead, and the restarts made the database slower, and the loop ran until someone read…
DevOps
The Rollback That the CDN Kept Serving for an Hour
The deploy was reverted in four minutes, and the incident lasted an hour, because the edge cache held the broken build and kept serving it to everyone the…