The flag was supposed to enable a new pricing display for five percent of traffic. I flipped it at 9:04 in the morning, watched the dashboard for twenty minutes, saw nothing, and assumed the rollout was just slow to propagate. At 12:20 a colleague restarted a pod to
The crash report said write EPIPE. The stack trace pointed at a line of code that was, by every reading, correct. It was writing the response to an open request. Writing to an open request is the entire job of a server. Except the request was not open anymore. The
A merchant reported that the cart count in the header was wrong. Not stale by a few seconds. Wrong by a whole session. One customer added three items and the header said eleven. Then it said two. It settled on the correct number only after a hard reload. Eleven was
A penetration test report landed with a finding rated critical. The tester had taken a valid token from their own low privilege account, changed the payload, set the header algorithm to none, stripped the signature, and been granted access to another tenant's data. We had read the advisories.
A service I worked on had a mean response time of 84 milliseconds. The dashboard was green for months. Then someone plotted the distribution and we found that one request in fifty took over two seconds, and those users were bouncing. The average was not lying. It was just describing
Every deploy produced a handful of 502s. Not many, never enough to trip an alert, and always dismissed as "the load balancer catching up". It was not the load balancer. It was our containers being killed mid request because nobody had written a signal handler. The window is
I avoided tcpdump for years because I assumed it meant learning to read hex dumps. It does not. Ninety percent of what I use it for is answering yes or no questions: did the packet arrive, did we respond, how long did the handshake take, who sent the reset. None
When I review a pull request I read the error handling before I read the feature. It is an odd order and it occasionally annoys people, but it comes from a simple observation: almost every incident I have been part of was an error path that had never been executed
Every runbook I have ever opened during an incident was wrong. Not catastrophically, not uselessly, but wrong in the specific way that costs fifteen minutes at three in the morning: the dashboard had been renamed, the command needed a flag that did not exist when it was written, and the
I have watched two engineers argue for twenty minutes about whether keep alive was enabled. One was looking at a Connection: keep-alive header. The other was looking at net.ipv4.tcp_keepalive_time. They were both right and they were talking about different layers of the stack. The naming
The ticket said search was broken. The user searched for a product they had bought before and got results for industrial fasteners. They do not sell industrial fasteners. Two hours later there were eleven tickets. Then a merchant called. The common thread was that every affected user was logged in,
The error looks like a certificate problem. It is almost never a certificate problem. error:1408F10B:SSL routines:ssl3_get_record:wrong version number Or, depending on which client you are using: curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number OpenSSL SSL_connect: SSL_ERROR_SYSCALL
Networking
A live feed that dropped on a strict interval in production and never locally. The interval was the whole clue and I ignored it for a day.
Docker
One changed byte early in a Dockerfile throws away everything after it. Here is how the cache key is computed and how to order instructions so it survives.
PostgreSQL
Dead tuples that never get reclaimed, a table twice the size of its data, and queries getting slower every week. Here is what blocks vacuum and how to see it.
Security
Tokens rejected as expired seconds after issue, or accepted long after they should have died. Both are the same bug seen from opposite ends.
Networking
Access-Control-Max-Age caches the preflight decision, including the one that said no. Here is why your fix looked like it did not work.
MicroVMs
You added the header at the proxy, your framework was already sending it, and the browser now rejects both. Duplicates are invalid, not additive.
MicroVMs
The bug only reproduces with real state. Copying the database loses the parts that matter. Cloning the whole machine keeps them.
DevOps
Lag climbing on one partition while the rest keep up tells you something specific. Here is how to read the numbers before you add consumers that cannot help.
MicroVMs
The question is not whether an autonomous agent will do something destructive. It is what the destruction can reach when it does.
JavaScript
A single page app that was fine for an hour and unusable by the afternoon. Three wrong hypotheses, then a heap snapshot showed forty thousand detached DOM nodes.
MicroVMs
You raised the body size limit and the upload still fails. There are almost always two limits, and the one you changed is rarely the one rejecting you.
MicroVMs
curl works on the box and hangs from outside. Four places the packet can die, and how to tell which one killed it without guessing.