The Cert Rotated at Midnight and the Pool Kept the Old Handshake

The certificate rotation ran at midnight, the automation reported success, and the monitoring showed a clean new certificate live on every node. At 00:00 sharp, a slice of the fleet began failing TLS handshakes, and the slice was exactly the set of clients that kept long lived connection pools. New clients connected fine. Pooled clients failed until their pools cycled, which took as long as the pool's idle timeout, so the incident's duration was a configuration value nobody had written down as an incident duration.

This is the autopsy of a rotation that was correct and still broke things, because the clients were not doing new handshakes. They were resuming old ones, and the old ones referred to a certificate the server no longer honoured.

This was a fleet of Node 22.14 clients with keep alive agents and a TLS terminator that rotated its certificate, and the failure is the collision between two caches that never talk to each other: the server's certificate store and the client's session cache.

The short answer

TLS resumption exists to skip the expensive handshake on reconnect, per TLS session resumption, and the handshake that never got faster. The client caches a session ticket or id tied to the certificate chain it saw. At rotation, the server swaps its chain and, typically, its ticket keys. A pooled client that resumes presents the old ticket, the server cannot or will not honour it, and the handshake fails, or the client, resuming against a connection whose peer certificate changed underneath, rejects the mismatch. Either way the pooled connection, the very optimisation meant to make reconnects cheap, becomes the failure path at the one moment every connection is supposed to converge on the new certificate.

The pool amplifies it, because a pool is exactly the set of connections that resume rather than re handshake, so the clients most invested in reuse are the clients most exposed to rotation, which is why the failing slice was the pooled half.

Why the rotation looked healthy

Every health check that performed a fresh handshake saw the new certificate and passed, so the rotation dashboard was green. The failing clients were the ones that did not perform a fresh handshake, and no health check exercises the resume path against a just rotated peer, because the health check's own connections are short lived. The monitoring measured the healthy population, the new handshakes, and missed the failing one, the resumptions, which is the sampling blindness from measuring TTFB when a CDN is answering for you, applied to handshakes: the metric covered the cheap path, not the cached one.

The causes, ranked

1. Resumed sessions outliving the certificate they vouch for

The primary cause. The client's session cache held tickets minted under the old chain, and the rotation did not invalidate them on the client, because the client was never told. Resumption trusts a token from the past, and rotation changes the present, and nothing reconciles the two.

2. Pools that never recycle on certificate change

The pool treats a connection as reusable until idle timeout or error, and has no notion of the peer's certificate rotating. A pool that does not watch the peer chain cannot drain itself at rotation, so it keeps lending out connections whose resumption context is stale.

3. Rotation without a drain window

The server retired the old ticket keys immediately, so in flight resumptions failed hard instead of being honoured for a grace period. A rotation that accepts the old ticket for a short overlap window converts the failure into a quiet full handshake, at the cost of a slightly longer grace for the old key.

The fix

The immediate fix restarted the pooled clients, which forced fresh handshakes and cleared the incident, and that restart is the confession that the pool had no rotation story.

The durable fixes were three. First, the client pool now watches the peer certificate fingerprint on each checkout, and on change drains the affected connections instead of reusing them, so a rotation becomes a coordinated re handshake rather than a failure storm. Second, the rotation gained an overlap window in which the server still honours the previous ticket key, so a resumption during the cutover falls back to a full handshake instead of an error, which is the dual credential window from rotating a database password without dropping a request, translated to TLS. Third, the health check now includes a resumed handshake against a warm session, so the resume path against a fresh rotation is exercised in the green dashboard, because the metric must cover the cache or it covers nothing.

What I would do differently

I would have treated the session cache as part of the rotation's blast radius, because a cache is always part of a rotation's blast radius, whether it holds passwords, plans or tickets. The rotation plan listed the servers it touched and never the clients that cache, and the clients were the ones that failed.

I would also have graphed handshake type, full versus resumed, per client, because the incident's signature, resumptions failing while full handshakes succeed, is visible in that split from the first minute, and the split is the diagnosis.

If it happens again

A rotation changes the present while the client's session cache vouches for the past, and pooled connections, which resume rather than re handshake, are exactly the population that carries the past into the rotation. Drain pools on peer certificate change, honour the old ticket for a short overlap, and exercise the resumed handshake in the health check.

The certificate was never the problem. The problem was a cache that outlived the fact it cached, which is the oldest defect in this archive, wearing a ticket, and the pool was the delivery mechanism.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud