The Failover That Promoted a Replica Hours Behind

The failover was, by every automated measure, a success: the primary died at 14:00, the replica was promoted within seconds, and the service reconnected. Then the orders placed between 11:00 and 14:00 began returning not found, because they had been written to a primary whose WAL the replica had not yet received, and the promotion made the lag permanent. The system did not lose data to the crash. It lost data to a replication gap that had been open for three hours, and the gap was invisible because the monitoring measured the wrong thing.

This is the anatomy of a failover that promotes a stale replica, and of the lag monitoring that is supposed to prevent it and quietly does not.

This was Postgres 16.3 with streaming replication and an automatic promotion, and the failure is a composition of two correct components: a replication that was asynchronous by design, and an alarm that watched the wrong end of the pipe.

Why asynchronous replication is a promise with a gap

Streaming replication ships WAL from primary to replica continuously, but asynchronously: the primary commits without waiting for the replica, which is exactly what makes the primary fast and available, and exactly what guarantees a gap. At any moment the replica is some distance behind, usually milliseconds, and the distance is the amount of committed data that exists only on the primary. Promotion converts that distance into loss, by definition. Asynchronous replication is therefore a contract: you accept losing up to the current lag in exchange for the primary's independence, and the size of the acceptable loss is a number the business should own.

The incident's first failure was that nobody owned the number, so the acceptable loss was whatever the replica happened to be behind by, which turned out to be three hours.

How a replica falls hours behind without an alarm

A healthy replica applies WAL about as fast as the primary produces it, so the lag stays small. The lag grows when the replica cannot apply, and the ways it cannot apply are specific.

Apply blocked by a long query or lock on the replica. A reporting query holding a conflicting lock on the replica stalls apply, because apply needs the same lock, and hot standby waits or cancels, and the waits accumulate into hours. The replica is serving reads beautifully while falling behind, which is the cruel part: the thing making it useful is the thing making it stale.

Apply slower than production. A replica on slower disk, or throttled CPU, applies WAL slower than the primary generates it during peak, and the gap opens precisely when the traffic is highest, which is when the failover is also most likely.

Network degradation between the pair. A lossy or saturated link slows the WAL stream while the primary, asynchronous, feels nothing.

A silent apply error. A conflict that cancels apply workers, or a misconfigured standby that stops fetching, leaves the replica frozen at an old LSN while it still answers health checks, because serving old data is not an error to a process that does not know it is old.

Why the alarm never fired

The monitoring watched the replica's health, is it up, is it serving, and the replica was up and serving, faithfully, three hours of stale data. The metric that mattered, the replication lag, was either not collected or collected with a threshold set to a value chosen for alert volume rather than for the acceptable loss, and a threshold of, say, one hour will happily page at hour one only if it is wired to page, and this one was wired to a dashboard.

This is the health check blindness from outage caused by a successful health check, at the replication level: the replica's aliveness said nothing about its currency, and currency was the property the failover was about to bet on.

The numbers that should be owned and alarmed

The lag is measured as the byte or time distance between the primary's commit position and the replica's applied position, and it must be alarmed as a budget: alert when the lag exceeds the acceptable loss, which the business names, not when it looks large. A lag budget of thirty seconds means a failover loses at most thirty seconds, and the alarm at thirty seconds is the contract being enforced.

The second number is the replica's apply state: is apply running, is it waiting on a lock, how long since the last applied WAL. A replica that is up but not applying is frozen, and the freeze is the emergency, because a frozen replica is a time machine pointed backwards.

The third is the promotion rehearsal: fail over in a drill, and measure what was lost, because the loss number is the lag budget's audit, and the drill is the only place the contract is tested, per a backup you have never restored is a rumour, and the shared property is the one worth fixing.

The fixes that change the contract

Where the loss budget is zero, the replication mode changes: synchronous replication makes the primary wait for the replica, closing the gap at the cost of the primary's availability when the replica fails, which is trading one failure mode for another, deliberately. Where the budget is small but nonzero, keep it asynchronous and enforce the budget with the alarm and with apply tuning that keeps the replica fast: keep long reporting queries off the standby that must stay current, or give reporting its own, staler, replica, so the failover candidate is not the one blocked by the analysts.

And the promotion logic should consult the lag before promoting, refusing or warning when the candidate is beyond the budget, because an automatic promotion that ignores the lag is an automatic data loss event with a trigger.

The rule I keep

Asynchronous replication is a contract that accepts losing up to the current lag, and a failover that promotes a replica hours behind is the contract executed, not broken. Name the acceptable loss, alarm the lag at that budget, watch the apply state for freezes, keep the failover candidate free of blocking reads, and rehearse the promotion to measure what the contract actually costs.

The replica was not lying. It was serving, quickly and politely, the past, and the only instrument that knew the difference was the one number nobody had turned into an alarm, which is the oldest failure in this archive: the metric exists, and the alarm does not.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud