I Rebooted the VM and the Kernel Did Not Change
A patched kernel that kept reporting the old version after every reboot. The reboot was working perfectly. My assumption about where the kernel comes from was wrong.
A CVE landed, the fix was in a newer kernel, and the patch had been rolled out. I rebooted the machine, ran uname -r, and got the old version back. Rebooted again. Same.
Nothing errored. reboot did exactly what reboot does. The machine went away, came back, and reported a kernel I thought I had replaced twenty minutes earlier.
The symptom, stated precisely
uname -r
# 6.11.0-19-generic
apt list --installed 2>/dev/null | grep linux-image
# linux-image-6.11.0-24-generic installed
sudo reboot
# ... wait ...
uname -r
# 6.11.0-19-generic <- unchanged
A newer kernel package was installed. The bootloader config named it. The reboot completed cleanly. And the running kernel was still the old one, with no message anywhere saying why.
Tested on Ubuntu 24.04, Firecracker 1.10.
The hypotheses that were wrong
Hypothesis one: the bootloader is picking the wrong entry
The obvious first guess when a kernel upgrade does not take. On a normal machine this is usually a GRUB default pointing at an older entry.
I went looking for it and found the problem with my own reasoning immediately: there was no GRUB config to inspect. No /boot/grub/grub.cfg, no menu entries, nothing.
I noted that as odd and, embarrassingly, moved on rather than pulling the thread.
Hypothesis two: the package installed but the initramfs was not rebuilt
Plausible. A kernel package that fails to regenerate its initramfs can leave a system booting the previous one.
sudo update-initramfs -u -k all
sudo reboot
uname -r
# 6.11.0-19-generic
No change. Also, /boot was nearly empty, which was the second time the filesystem had told me something and I had not listened.
Hypothesis three: the image is being restored from a snapshot on every boot
This one was closer to the truth and still wrong. If the disk were reverting, my package installs would have vanished too. They had not. apt list --installed still showed the new kernel package after every reboot.
So the disk was persistent and the kernel was not coming from it. Those two facts together are the whole answer, and it took me a while to put them side by side.
The breakthrough
The kernel was not on the disk at all.
A Firecracker microVM does not boot the way a physical machine or a traditional VM does. There is no BIOS, no bootloader, no GRUB. The hypervisor is handed a kernel image and a root filesystem, and it jumps straight into the kernel. That is a large part of why boot times are measured in milliseconds rather than tens of seconds: the entire firmware and bootloader phase does not exist.
The consequence is the thing that had been confusing me for an hour. The kernel comes from the host, not from the guest disk. When you run reboot inside the guest, the guest kernel goes through its shutdown path and the VMM starts it again, using the same kernel image it was given at launch. Your linux-image package sits on the disk being ignored, because nothing in the boot path ever reads it.
reboot from inside cannot change the kernel. It is not broken and it will never tell you.
What does change it is a cold restart, where the hypervisor process itself is stopped and relaunched, so it picks up the current kernel image from the host and boots the same disk against it.
# from outside the guest
krova cubes restart my-api
The disk is preserved. Only the kernel changes. On Krova Cloud, where I run this, a Cube that is already powered off picks up the refreshed kernel the next time you start it, so the only case that needs attention is a long running machine that has never been cold restarted.
What I changed
Stopped treating uname -r as a check on the guest. On a microVM it reports a property of the platform. The right question is not "did my package install" but "has this machine been cold restarted since the host kernel changed".
Added the check to our runbook explicitly.
# how long has this instance been running on its current kernel
uptime -s
uname -r
An instance whose start time predates the platform kernel update is running the old one, regardless of what is installed on its disk.
Separated the two update paths in my head, because they genuinely are separate.
| Layer | Source | How it updates |
|---|---|---|
| Guest userspace | Your disk | apt and unattended upgrades, no reboot needed |
| Guest kernel | The host, at boot | Cold restart from outside the guest |
Package security updates land daily without interrupting anything. The kernel is on a different mechanism entirely, and conflating them is what cost me the morning.
What I would do differently
Follow the anomaly instead of noting it. An empty /boot and a missing GRUB config are not minor curiosities, they are the answer. I saw both within the first ten minutes and kept testing my original theory. This is the same failure I keep writing about: the wrong assumption underneath the bug was that a kernel is something a machine loads from its own disk, and everything I tried was consistent with that assumption being true.
Ask where a thing comes from before asking why it did not change. I spent the whole morning on the update mechanism and none of it on the supply chain. One question, "where does this kernel actually come from", would have ended it.
Notice when the absence of an error is information. No warning, no failure, no log line. A silent no op almost always means the operation you think you are performing is not the operation the system is performing. That is worth treating as a signal in itself, the same way an error message that says nothing useful still narrows the search if you read what it does not say.
There is a security angle worth stating too. Because the kernel is supplied by the host, a compromised guest cannot pin itself to a vulnerable kernel. It gets whatever the host hands it on the next cold start. The property that confused me for an hour is the same property that makes the boundary hold, which is a reasonable trade once you know it exists.
