Bug Hunt
The Outage Caused by a Successful Health Check
Every replica restarted at once during a brief database blip. The health check was working exactly as designed, and the design was wrong.
CrashLoopBackOff, OOMKilled, DNS and everything else that goes wrong between a healthy container and a healthy pod.
Bug Hunt
Every replica restarted at once during a brief database blip. The health check was working exactly as designed, and the design was wrong.
Docker
Your image builds on a Mac and dies in production. What the error means, how to check an image's architecture, and how to build multi arch properly.
Bug Hunt
p99 jumped on one service. The database was fine, every dependency reported healthy, and the flame graph pointed at nothing. That turned out to be the clue.
Kubernetes
Exit code 137, a memory graph that looks perfectly healthy, and no application error. Here is why your monitoring missed it and how to find the real number.
Kubernetes
CrashLoopBackOff is not an error, it is a symptom with about nine distinct causes. Here is how to identify which one you have in under three minutes.
Kubernetes
nslookup resolves and your application cannot connect. The two use different resolvers, and the difference is where the bug lives.
Docker
Non root containers break in predictable ways. Here is how UID mapping, volume ownership, and read only filesystems produce the same error.
Docker
Distroless and scratch images are great for security and miserable when something breaks. Here is how to investigate a container that ships nothing.
Kubernetes
Two statuses, one underlying problem, and about six causes. Here is how to read the event message and fix the right thing.