> ## Content Index > Fetch the complete content index at: https://debugly.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Rotating a Database Password Without Dropping a Request - URL: https://debugly.dev/secrets-rotation-without-downtime/ - Published: 2026-09-29T11:30:00.000Z - Updated: 2026-10-10T13:48:22.000Z - Description: Applications cache credentials at boot, so a password change is a rollout problem, not a config change. The dual credential window is the only rotation that… - Author: Rohit Bhadani - Tags: DevOps, Security, Debugging The rotation was scheduled, the password was changed, and at the next deploy window the service began failing authentication in a slow, partial way: new pods healthy, old pods failing, connections flapping, because the old pods held the old password in memory from boot and would until they died. The rotation had treated a credential as a value in a store, and the system had treated it as a fact cached at startup, and between those two models, requests fell. I keep a private list of these, because the incidents that hurt are always the ones somebody assumed were harmless. A credential rotation is a distributed systems problem, because the secret lives in many places at once: the database, the secret store, and the memory of every running process that read it at boot. Changing one copy at a time, in the wrong order, produces a window where some holder has a secret no authority accepts, which is an outage with a security label. This was Postgres 16.3 with a password authenticated role, a secret manager, and a rolling deployment, and the sequence below is the one that never drops a request. ## Why the naive rotation fails The naive sequence is: update the secret store, then the database, or the reverse, and let the pods pick it up "on restart". Each ordering has a fatal window. Change the database first, and every running pod holds a now invalid credential, failing until restarted, which is the incident. Change the store first and the database later, and pods restarted in between fetch a credential the database does not yet accept, which is the same incident arriving via the deploy. The root cause is that applications read credentials once, at boot, and the rotation's duration is therefore bounded by the fleet's rollout time, not by the operator's speed. Any sequence that has only one valid credential at a time will disagree with some holder for the length of a rollout. ## The dual credential window The fix is to make two credentials valid at once, for a bounded window, so that every holder, old or new, is always right. Postgres supports two passwords on a role during the transition through the mechanism of updating the credential while the old one still verifies, in practice by adding the new credential before removing the old, which for password auth means the verifier accepts both during the window. Where the store is a token or an API key, the provider almost always offers a create then revoke flow, a second active key, precisely because the single key rotation is the outage everyone has had. The sequence is then: create the new credential at the authority, so both are valid. Update the secret store to the new value, so new boots fetch the new one. Roll the fleet, so every holder now uses the new one, while old holders still succeed on the old. Verify no holder uses the old, from the authority's audit of which credential authenticated. Revoke the old. The window closes, and at no point did any holder hold an invalid secret. ## The restart assumption and the pool The dual window assumes every holder eventually re reads the credential, which is true for processes that restart, but the fleet includes holders that do not: long lived workers, cron that reads at schedule start, and connection pools that authenticate once and lend the connection to many clients. The pool is the subtle one. PgBouncer and its kin authenticate to the database with their own credential at their own boot, and clients authenticating through the pool may be checked against the pool's cached credentials rather than the database's current ones, so the pool is a holder with its own rollout, and it must be restarted inside the window like any other holder, or configured to re authenticate, or the window must outlive it. This is the pool's session state lesson from [PgBouncer transaction mode](https://debugly.dev/pgbouncer-transaction-mode-prepared-statements/), applied to credentials. The audit step, asking the authority which credential authenticated which holder, is what makes the revocation safe, because it converts "probably rolled" into "observed rolled", and revoking on observation rather than on hope is the difference between a rotation and a gamble. ## Making rotation routine instead of ceremonial The reason rotations are rare is that they are scary, and they are scary because they are manual sequences with fatal windows, and the fix for scary manual sequences is to make them boring automated ones. Rotate far more often than compliance requires, on a schedule, with the dual window automated and the audit asserted, because a rotation that runs weekly is a rotation whose failure modes are found weekly, in small recoverable form, instead of annually, during the incident that forced it. The annual rotation is the dangerous one, precisely because it is unpractised, which is the restore drill argument from [a backup you have never restored is a rumour](https://debugly.dev/backups-you-have-never-restored/), applied to credentials. And surface the age of each credential, and the last observed use of each old credential, on the same dashboard as the backup's restore age, because both numbers are the age of an untested safety property, and both rot quietly. ## The takeaway A credential lives in every holder's memory at boot, so rotation is a rollout, and a rollout needs a window where old and new are both valid. Create the new, publish it, roll the fleet including the pools, observe the old go quiet, then revoke it, and automate the sequence until it is boring. The single credential rotation is a distributed system being asked to change state atomically, which is the one thing it cannot do, and the dual window is the admission that makes the change safe. The leak side of the same secret, where the credential escapes into history, is [deleting the commit did not delete the secret](https://debugly.dev/secrets-in-git-history-are-still-there/), and the shared property is the one worth fixing.