Six Months of Running MicroVMs in Production

Disclosure: I run Krova Cloud, which is built on Firecracker. This is an engineering account of what we learned building the platform, and everything here is now solved and running in production.

We have been running Firecracker microVMs in production for a while now. The technology delivers exactly what it promises. The engineering that turns a hypervisor into a platform is the interesting part, and that is what this post is about.

What worked immediately

Boot times held up. Sub second cold boots in production, consistently, at load. The architecture genuinely delivers this and it was not a benchmark artifact.

Isolation did its job. No cross tenant incidents. The property we built on turned out to be the property we got.

The VMM is small and stable. Firecracker itself has been the least troublesome component in the stack, which is not what I expected from the newest piece.

What surprised me

Memory is the constraint, not CPU

I planned capacity around CPU. Memory is what actually limits density, because each microVM reserves its allocation rather than sharing page cache with neighbours the way containers do.

A host with plenty of idle CPU can be completely full on memory. We rebuilt our scheduler around memory as the primary constraint, which is why every Cube reserves its RAM and disk 1:1 against real hardware. Customers get exactly what they provision, and a noisy neighbour cannot take capacity someone else paid for.

Balloon devices let the host reclaim memory a guest is not using. They need careful tuning, because reclaiming too aggressively causes guest OOM kills. We tuned them conservatively and they have run cleanly since.

Snapshot restore is harder than the demo

Restoring from a snapshot is genuinely fast. Then you meet the three problems that no benchmark shows you.

Entropy. Every restore from the same snapshot has identical random state. Any key generated shortly after restore is predictable. The guest has to reseed, and virtio-rng notification is the supported mechanism, and you have to actually wire it up.

Time. The guest resumes believing it is whenever the snapshot was taken. TLS validation fails against certificates that are not yet valid, tokens appear unexpired when they are not, and log timestamps are nonsense until something corrects the clock.

Network state. Connections in the snapshot are dead. The peer forgot about them long ago.

Each of these is a correctness problem that only appears in production. We handle all three in the platform now, which is why snapshot restore and cloning is something customers can rely on rather than something they have to engineer around.

Networking was the largest engineering cost

Not the hypervisor. The networking.

Every microVM needs a tap device, an IP, firewall rules, and a route. At scale that is thousands of interfaces per host, and Linux networking has some interesting behaviour at that scale.

Specific things that bit us: conntrack table exhaustion under high connection churn, which manifests as random connection failures with no useful error; ARP table limits on the host, defaults are lower than you would want; and IPv6 neighbour discovery doing surprising things when interfaces churn rapidly.

None of these are Firecracker problems. They are what you inherit when you take responsibility for the network layer, and it was the bulk of the work. It is also the reason Cubes are isolated from each other in the kernel rather than by a firewall rule, and why a Cube has no inbound route until you open a port.

The guest kernel became a real maintenance burden

A minimal custom kernel is a thing you now own. Security patches, testing, and the discovery that a driver you excluded is needed by a customer workload you did not anticipate.

We treat it as an ongoing responsibility rather than a one time configuration decision. That is why the kernel is supplied by the host at boot rather than from the guest disk: we can ship a patched kernel to every Cube, and a compromised guest cannot pin itself to an old one.

The incidents that taught us the most

The conntrack thing, twice. First occurrence took hours because the symptom was intermittent connection failures spread across unrelated workloads with no pattern in any application log. The table filling produces silent packet drops rather than an error. nf_conntrack: table full, dropping packet in dmesg is the only signal and we were not watching dmesg.

That is a textbook absence-of-signal failure: the system was telling us and nobody had subscribed.

A snapshot with a stale clock caused a TLS outage for restored instances. Certificates were rejected as not yet valid. Diagnosis was slow because the certificate was obviously fine when checked from anywhere else.

Disk I/O contention between neighbours. Reserved memory and CPU does not reserve IOPS. One workload doing heavy sequential writes could degrade others on that host. We added per VM I/O throttling, which completed the isolation story: CPU, RAM, disk capacity and now disk throughput are all bounded per Cube.

A metadata service race where a VM booted faster than its configuration was written. Sub second boot exposed an ordering assumption elsewhere in the system. We made provisioning strictly ordered, and the boot speed that caused it is now one of the things customers rely on most.

What I would tell someone considering Firecracker

Build on it if isolation matters to you. If you are running code you did not write, whether that is AI agents, customer workloads or CI from forked pull requests, a kernel boundary is the property you need and Firecracker is the right foundation. The decision framework matters more than the technology.

Or skip the build entirely. Everything in this post is engineering we did so customers do not have to. A Cube gives you the same isolation with the scheduling, networking, snapshot correctness and kernel maintenance already handled.

Budget for networking, not virtualisation. Firecracker is well documented and mostly just works. The surrounding infrastructure is where the time goes, and it is not the part anyone writes tutorials about.

Instrument the host, not just the guests. Conntrack, ARP, file descriptors, the tap device count, dmesg. These are the things that break at scale and they produce no application level signal at all.

Test snapshot restore properly. Not "does it restore" but "does a restored instance behave correctly ten minutes later". Time, entropy, and network state all need explicit handling.

Reserve headroom. Memory reservation means overcommit is not available as a safety valve. Running hosts at 95 percent leaves nothing for a workload that legitimately needs to grow.

What I got right

Building on Firecracker rather than writing a VMM. Obviously. But I want to note that the temptation to customise the VMM existed and resisting it was correct. Every problem we had was solvable outside it.

Isolation as a floor rather than a tier. We never shipped a cheaper shared kernel option. It would have been commercially tempting and it would have undermined the only thing that makes the product coherent. Isolation is not an upgrade you buy on Krova, it is the floor.

Watching dmesg. After the first conntrack incident. Kernel messages are a genuinely valuable and routinely ignored signal source when you operate at this level.

The thing that has not changed

The debugging method is the same one I have used since I was keeping shared hosting customers online in 2019.

The tools moved down the stack, from application logs to kernel traces. The method did not: form a hypothesis about which assumption is false, find the cheapest experiment that would prove it wrong, run it.

The conntrack incident was solved by noticing that failures were spread across unrelated workloads, which meant the cause was shared rather than specific. That is the same reasoning as checking shared infrastructure when one service is slow and its dependencies are healthy, applied one layer lower.

Which is roughly what I keep finding. The domain changes completely and the method transfers entirely.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud