Networking
Cloudflare Error 526 and the Redirect Loop: Two Failures, One Setting
Flexible gives you an infinite redirect. Full strict gives you a 526. The documentation says Full or higher, and higher is the one that breaks.
Node.js
EMFILE: Too Many Open Files, and Why Raising the Limit Is Usually Wrong
The quick fix is raising ulimit. The real question is what is holding thousands of descriptors open and never closing them.
MicroVMs
I Rebooted the VM and the Kernel Did Not Change
A patched kernel that kept reporting the old version after every reboot. The reboot was working perfectly. My assumption about where the kernel comes from was wrong.
Bug Hunt
The Outage Caused by a Successful Health Check
Every replica restarted at once during a brief database blip. The health check was working exactly as designed, and the design was wrong.
Docker
exec format error: The ARM64 and AMD64 Mismatch Explained
Your image builds on a Mac and dies in production. What the error means, how to check an image's architecture, and how to build multi arch properly.
Engineering Culture
Most Postmortems Are Theatre and Here Is What Makes One Useful
A document nobody reads, listing action items nobody does, produced to satisfy a process. Here is the version that actually changes something.
DevOps
What to Do in the First Ten Minutes of an Incident
The instinct is to start investigating. The first ten minutes are better spent on four other things, and doing them makes the investigation faster.
Bug Hunt
A 400ms Latency Spike That Turned Out to Be DNS
p99 jumped on one service. The database was fine, every dependency reported healthy, and the flame graph pointed at nothing. That turned out to be the clue.
Networking
Your Webhook Handler Will Be Called Twice and You Should Plan For It
At-least-once delivery means duplicates are normal operation, not an edge case. Here is how to build a handler that survives them.
Deep Dive
How a MicroVM Boots in Under a Second
Traditional VMs take thirty seconds. Firecracker takes 125 milliseconds. The difference is everything it refuses to do.
DevOps
Your CI Pipeline Has More Access Than Your Engineers Do
A build system with production credentials and the ability to run arbitrary code from a pull request is the most under-secured part of most stacks.
Kubernetes
OOMKilled: Why Your Container Died and Your Metrics Showed Nothing
Exit code 137, a memory graph that looks perfectly healthy, and no application error. Here is why your monitoring missed it and how to find the real number.