DevOps
The Rollback That the CDN Kept Serving for an Hour
The deploy was reverted in four minutes, and the incident lasted an hour, because the edge cache held the broken build and kept serving it to everyone the…
I write about why software breaks and how people figure it out. Founder of Krova Cloud, building isolated microVMs on our own hypervisor. Years of Shopify theme work and shared hosting before that. If I have not reproduced it, I do not publish it.
DevOps
The deploy was reverted in four minutes, and the incident lasted an hour, because the edge cache held the broken build and kept serving it to everyone the…
Code Review
The loader fell back to a built-in value when the key was absent, so the service ran with the wrong setting for a week, and the startup log said the…
Tooling
The kernel already exports every counter you would install an agent to collect. The files under /proc are the metrics endpoint, and reading them is the…
Ecommerce
The customisations lived in the theme files rather than an override, so the update replaced them, and the tracking snippet and the warranty copy disappeared…
Deep Dive
The pod was serving fine until Kubernetes killed it, repeatedly, because a liveness probe shared the event loop with the work.
Security
The task was to summarise a repository, and the token the agent ran with could delete its branches. The scope was copied from a deploy credential, and the…
Performance
The invoice tripled with no traffic change, because the prompts tripled, and nobody was reading tokens as a metric.
Error Autopsy
The probe hit the public hostname and the edge returned a cached 200, so the platform believed the service was healthy while every pod behind it was…
Debugging
The agent broke the build once, and no rerun would do it again, because the failure was a draw from a distribution, not a function of the input.
Shopify
The srcset was correct and the browser still downloaded the largest file. The sizes attribute was lying, and almost every theme I audit gets it wrong.
DevOps
The pipeline gained an autonomous step and, with it, a workload that could loop, spend and touch production at three in the morning.
Tooling
The agent was given a shell and, eventually, a mistake. The boundary that made the mistake cheap was not the model's judgement but the sandbox: disposable…