Everything Broke the Day We Doubled the Instance Count

The team added capacity to absorb a traffic spike and, within minutes of the scale out, the entire service began failing with database connection errors. The direction of causality read as impossible: more instances caused an outage. It was not impossible. The connection pool was sized per instance, and the database's maximum connections is sized per database, and the two budgets had never been multiplied together until the day the multiplication happened.

This is the arithmetic bug of horizontal scaling, and it is worth its own autopsy because every number involved was correct in isolation and reviewed as correct, and the product of them was the incident.

This was a service with ten instances, each running a pool of ten connections to a Postgres 16.3 configured for a hundred and twenty connections, scaled to twenty instances for the spike.

The symptom, precisely

Before the scale out, the database ran with about a hundred of its hundred and twenty connections in use, headroom that looked comfortable. After the scale out, two hundred connections were attempted, the database refused new ones at its limit, and every instance began failing to acquire connections, including the instances that had been running fine before, because the new instances' attempts queued and timed out behind the existing ones. The outage was total, not partial, which is the signature of a shared resource exhausted rather than a bad node.

The hypotheses that died

The new instances are misconfigured

First I compared the new instances' configuration to the old ones. Identical. The pool size, the timeouts, everything matched.

It died because the new instances were not different. They were the same, and the sameness was the problem.

The database is failing

Next I suspected the database itself, a crash or a disk fault. It was healthy, refusing connections politely at its configured limit, which is correct behaviour under an impossible demand.

It died because the database was doing exactly what it was configured to do. The configuration was the defect.

A connection leak

Then I looked for a leak, connections acquired and never returned. The count per instance matched the pool maximum exactly, no leak. Every instance was holding precisely what it was configured to hold.

It died because there was no leak. The honest sum of honest per instance settings exceeded the shared budget, which is a harder truth than a leak, because there is no bad line to delete, only a budget that was never multiplied.

The breakthrough

The breakthrough was not a discovery but a multiplication. Ten instances times ten connections equals one hundred, within the hundred and twenty limit with headroom that looked like safety. Twenty instances times ten equals two hundred, nearly double the limit. The headroom that made ten instances comfortable was the same headroom that made the doubling catastrophic, because it had never been budgeted as headroom. It had been treated as slack, and slack that is not a decision is a load waiting for a scale event.

The per instance pool size is a local decision, made by whoever tuned one instance, and the database maximum is a global decision, made by whoever provisioned the database, and nothing in the system connects the two, so the product is nobody's number until it is everybody's incident.

The fixes

Budget the product, not the parts. The global maximum must be divided among the maximum instance count the autoscaler can reach, with headroom for the database's own needs, and the per instance pool set to that quotient. The autoscaler's maximum is part of the pool configuration, because it is the multiplier, and a pool sized without the multiplier is sized for a fleet that does not exist.

Add a pooler between the fleet and the database. A pooler such as PgBouncer in transaction mode lets many instances share few backends, decoupling the instance count from the connection count, which is the structural fix and the reason poolers exist at scale. The trade is the session state semantics from PgBouncer transaction mode, which is a known price rather than a surprise.

Fail fast and shed load on acquisition. When a connection cannot be acquired, the request should fail quickly with a clear error rather than queue behind a timeout, because a long acquisition timeout converts a capacity shortfall into held requests and piled threads, which is the fork and task count story from fork retry resource temporarily unavailable at the application layer.

Alert on the database's connection utilisation. The ratio of used to maximum connections, per the database, is the metric that predicts this incident, and alerting at eighty percent gives time to act before the limit is the outage. The per instance pool metrics will never show it, because each instance is fine. The global number is the only one that tells the truth.

What I would do differently

I would have done the multiplication in the capacity review, before the autoscaler existed, because the autoscaler's maximum instance count is a commitment to a connection count, and commitments should be priced when made. The scale out was not the cause. It was the first time the configured fleet, the configured pool and the configured database were all true at once.

If it happens again

A per instance resource setting times the autoscaler's maximum instance count is a global commitment, and the database's limit is the budget it must fit. Price the product when you size either factor, put a pooler between the fleet and the database when the product outruns the budget, and alert on the global utilisation, because the per instance numbers are each correct and collectively impossible.

The queueing tail this produces while the limit is approached, before it is hit, is the p99 story in your average latency is fine and your users are not.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud