What Running Hosting Infrastructure Taught Me About Debugging

Disclosure: I run Krova Cloud, an infrastructure company. This post is about how the work changed my thinking.

I started my career in 2019 at a shared hosting company in Surat, keeping other people's websites online. It was not glamorous work. It was tickets, phone calls, and a lot of evenings spent on problems that were not mine.

It also taught me more about debugging than anything since, and I have been thinking about why.

The feedback loop is immediate and it is a person

When you build a product, a bug becomes a ticket. Somebody triages it, it enters a backlog, and you see it as a line item three days later with the emotional content removed.

In hosting, the bug is a person on the phone whose business is offline right now.

That changes what you optimise for in a way I did not appreciate until I left. You stop caring about the elegant explanation and start caring about the fastest path to service restored. You learn to separate the two operations that most engineers merge: stopping the bleeding and understanding the cause. They are different jobs with different time pressures, and doing them in the wrong order costs somebody money.

The habit that came out of it: when something breaks, my first question is now "what is the smallest action that restores service" rather than "what is happening here". The second question still gets asked, and it gets asked after, from a position of not being on fire.

Most engineers I have worked with since do this backwards. They investigate first because investigating is more interesting, and they are often right that it is fine to do so. But when it is not fine, the cost is entirely borne by somebody else.

You cannot reproduce anything

In application development, the reproduction is the goal. Get it failing on your machine, then iterate.

In hosting, most problems are unreproducible by construction. It is happening on one customer's site, with their plugins, their traffic, their data, at a moment that has passed. You cannot ask them to wait while you set up an identical environment, and you frequently cannot look at their code.

So you get good at inference from indirect evidence. Server logs, resource graphs, the timing of the report, what changed on the box, what else broke at the same moment. You develop a strong sense for which signals are reliable and which are the customer's interpretation of a signal.

That skill transferred directly. The DNS latency spike I wrote about was solved by exactly this kind of reasoning: nothing was reproducible, the profiler showed nothing, and the answer came from noticing that flat CPU with rising latency means the time is being spent waiting rather than computing. That deduction is a hosting habit.

The population, not the instance

The single most valuable thing I learned.

When one customer reports something, you have almost no information. When you check whether it correlates with anything, you often have the answer immediately. Same PHP version? Same server? Same plugin? Same time window? Started after the same maintenance?

One report is an anecdote. The distribution of reports is data.

I keep relearning this. The bug that only happened on Tuesdays sat open for five months with four engineers examining individual failures. It collapsed in ninety seconds when somebody finally aggregated the failure timestamps by weekday. Nobody had done it because the ticket said "no pattern identified", and everybody inherited that claim without re-testing it.

If a problem is intermittent, stop looking at the instance. Count.

Blast radius is a design property

Shared hosting fails in a specific way: one customer's problem becomes everybody's problem. A runaway process, a plugin with an infinite loop, a site that gets featured somewhere and receives fifty thousand visitors. The box slows down and two hundred unrelated websites degrade.

You spend a lot of time on containment, and eventually you stop thinking of isolation as a security feature and start thinking of it as an availability feature. The question stops being "can this customer read another customer's data" and becomes "can this customer's bad afternoon become another customer's bad afternoon".

That framing is most of why I now build what I build. Not because shared kernels are insecure in the abstract, though that is also true, but because I watched noisy neighbours take down people who had done nothing wrong, repeatedly, for years.

It also changed how I review application code. I now read for blast radius: if this fails, what else fails? A retry loop with no backoff is not just a bug in one service, it is a way of turning a dependency's brief problem into your dependency's sustained outage.

Silence is not health

Hosting teaches you that no alerts means your monitoring is incomplete, not that everything is fine.

You learn the categories of failure that generate no signal at all. Backups that run successfully and produce unrestorable archives. A certificate that expires on a subdomain nobody checks. A cron that stopped firing four months ago. Disk filling at a rate that gives you six weeks of warning, which nobody looks at because it is not urgent yet.

The habit: alert on the absence of expected events, not only on the presence of errors. A job that should run hourly and did not is a stronger signal than a job that ran and errored, because the error at least tells you something is alive.

This is the same reasoning behind logging the negative case when print debugging. Silence is ambiguous. It could mean the code did not run, or it ran and found nothing, and those are very different.

Most incidents are changes

Somewhere in my second year I started asking "what changed" before anything else, and my time to diagnosis dropped substantially.

Systems that were working and are now not working have usually been changed. A deploy, a config edit, a package update, a certificate rotation, a DNS change, a provider's maintenance window, or somebody's manual fix from three weeks ago that finally interacted with something.

The trap is scoping the question too narrowly. In the DNS incident, I searched for deploys to my own service and found none, concluded nothing had changed, and lost an hour. The trigger was another team's rollout seven minutes earlier. "What changed" has to mean everything in the blast radius, not everything you own.

You will be woken up by your own decisions

The most useful discipline of all, and the hardest to transfer to people who have not experienced it.

When you operate what you build, every shortcut has a personal cost with a delay attached. The missing timeout, the retry without backoff, the log line that would have made this obvious, the error message that says only "failed". You meet all of them again at two in the morning, and you remember.

Engineers who have never operated their own code tend to under-invest in exactly the things that make operation possible: structured logs, actionable errors, metrics on the right quantities, runbooks. Not from carelessness, just from never having received the feedback.

If you have never been on call for something you wrote, I would push for it. It is the fastest available education in why error messages are a user interface and why observability is not overhead.

The thing that has stayed constant

I have moved from shared hosting to Shopify themes to microVMs, and the debugging method has barely changed.

Form a hypothesis about which of your assumptions is false. Find the cheapest experiment that would prove it wrong. Run it. Repeat.

The tools change completely. Server logs, then Chrome DevTools and Liquid render times, now kernel traces and hypervisor metrics. The method does not, and the people who are good at it in one domain are usually good at it in the next, which suggests the transferable part is the reasoning rather than the tooling.

Which is roughly the thesis of this whole blog: every bug is a story about a wrong assumption, and the job is working out which one.