Your Database Got Slower and Nothing in Your Code Changed

A Postgres instance that had been steady for months started producing p99 query times that doubled for twenty minutes at a time, at no consistent hour, with no deploy, no traffic change and no growth in the dataset.

Everything inside the machine looked fine, which is the specific frustration this post is about. When your own metrics are healthy and the symptom is real, the cause is usually one layer below where you are looking.

First, prove it is not you

Before blaming anything external, rule out the ordinary causes properly. In order of how often they are actually responsible:

-- 1. a plan that changed under you
SELECT query, calls, mean_exec_time, stddev_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time * calls DESC
LIMIT 10;

-- 2. bloat quietly accumulating
SELECT relname, n_live_tup, n_dead_tup,
       round(100.0*n_dead_tup/NULLIF(n_live_tup+n_dead_tup,0),1) AS dead_pct
FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 10;

-- 3. checkpoints thrashing
SELECT * FROM pg_stat_bgwriter;

If stddev_exec_time is large while mean_exec_time is stable, the same query is sometimes fast and sometimes slow, which points away from the plan and towards resources. A changed plan is consistently slow, not intermittently slow. I covered that case in why the planner stopped using your index, and it was not this.

Tested on Postgres 16.3, Linux 6.8.

The metric that actually tells you

Inside a guest, the useful signal is steal time and I/O wait, not CPU utilisation.

# st is the column that matters
vmstat 1 20

# same figure, per core
mpstat -P ALL 1 5

# io wait, and whether the device is saturated from your side
iostat -x 1 5

%st is time the hypervisor scheduled someone else while your vCPU was runnable. Your process wanted to run and could not. It is the clearest evidence that the constraint is outside your machine.

A sustained %st above 5 percent is worth investigating. Above 15 percent your performance is being decided by someone else's workload.

The equivalent for storage is await climbing in iostat while your own IOPS stay flat. You are waiting longer for the same amount of I/O, which means the device is busy with traffic that is not yours.

What overselling actually is

Most budget hosting overcommits. The provider sells more RAM and more disk than the host physically has, betting that customers will not use their full allocation simultaneously. Usually that bet pays off, which is why the pricing works.

When it does not pay off, the mechanisms are these:

Memory. With overcommitted RAM, the host reclaims from guests under pressure. Ballooning shrinks your available memory from underneath you, and the page cache is the first casualty. For a database this is severe, because Postgres depends heavily on the OS page cache holding hot data. Losing it converts memory reads into disk reads, and your query times double without a single row changing.

Disk. Thin provisioning means your 40 GB is an accounting entry, not 40 GB of reserved blocks. It also means IOPS are shared. Reserved CPU and RAM do not reserve throughput, so a neighbour running a heavy sequential write can degrade everyone on the device.

CPU. Oversubscribed vCPUs show up directly as steal time.

The uncomfortable part is that none of this is visible from inside the guest except as symptoms. You see the effect, never the cause, and you cannot fix it from where you are standing.

Why reservation changes the failure mode

The alternative is a platform that reserves what you provision. On Krova Cloud, where I work on this, RAM and disk are held 1:1 against real hardware for as long as the machine exists. A 4 GB Cube holds 4 GB of host memory to itself, and a 40 GB disk occupies 40 GB of real storage whether it is full or not.

That is a deliberate trade. It costs more per host to run and it removes the possibility of the failure above. Your page cache is yours, your working set stays warm, and no neighbour can take capacity you paid for.

Worth being precise about what reservation does not cover: reserved memory and CPU do not automatically reserve disk throughput. That needs separate per instance I/O throttling, which we added after seeing exactly the neighbour effect described here. If you are evaluating any provider on this, ask specifically about IOPS as well as capacity, because the two are frequently conflated in marketing copy.

Making the invisible visible

Whatever platform you are on, instrument for this so the next occurrence is a lookup rather than an investigation:

node_cpu_seconds_total{mode="steal"}      alert above 5 percent sustained
node_disk_io_time_seconds_total           watch await, not just IOPS
node_memory_MemAvailable_bytes            a drop with no process growth means the host reclaimed
node_vmstat_pgmajfault                    major faults mean you lost page cache

Major page faults are the one people miss. A database that was serving from cache and is suddenly hitting disk will show a jump in pgmajfault before anyone notices the latency, which makes it an early warning rather than a post mortem.

Correlate those with your query latency on one dashboard. When p99 rises and steal time rises together, the conversation with your provider is a short one because you have the evidence. When p99 rises and steal time is flat, the problem is genuinely yours and you should go back to the query plans.

What to do about it

Short term. Move the instance. On most platforms a stop and start relocates you to a different host, which often resolves it immediately and also tells you something: if the problem follows you, it was never the neighbour.

Medium term. Give the database more memory headroom than it needs, so reclaim has slack to take before it reaches your working set. This is a mitigation, not a fix.

Long term. Run databases on reserved resources. A database is the workload where overcommitment hurts most, because its performance model assumes memory it has been promised is memory it has. Everything else can usually tolerate a bad twenty minutes.

Always. Alert on steal time and major page faults. The failure I have described is invisible for exactly as long as you are not measuring it, which is the same absence of signal problem that turns a slow degradation into a surprise incident.