Reviewing a Cache Layer for the Four Ways It Will Hurt You

The cache passed review the way caches always do. The read path returns the cached value when present and the fresh value when not, and the write path stores the fresh value. In the single reader narrative of a review, that is correct. In production, the four questions nobody asked produced the four incidents, and they are the same four incidents in almost every cache layer I have ever debugged.

So I review a cache by asking what happens in the four states, cold, hot, poisoned and mid deploy, and this is the checklist, with the reasoning behind each.

This is Redis 7.2 and an in process cache in Node 22.14, but the questions are storage independent.

1. What happens on a cold cache

The cold cache is the state after every deploy and every failover, and it is the moment the cache provides no protection at all, so the backend receives the full traffic with no help. The review question is whether the backend survives that moment.

If the answer is no, the cache has become load bearing, which inverts its role from optimisation to dependency, and a cache restart becomes an outage. The fixes are capacity for the cold state, a warm up path, or staggered invalidation so the cache never goes fully cold at once. The review should name the cold state traffic and the backend's headroom, because that arithmetic is the only thing that decides whether a failover is an event or an incident.

2. What happens on a hot key

A single key that many readers want is the second state, and the danger is the stampede. When the hot key expires, every waiting reader misses at once and every one of them hits the backend with the same expensive computation, which is the cold cache problem concentrated on one key.

The review wants to see the stampede handled, and the standard shapes are a lock or single flight so one reader computes and the rest wait for its result, or a stale while revalidate policy so readers take the old value while one refresh happens. A cache with a TTL and no stampede control is a stampede schedule, and the review should ask what the code does at the exact moment a popular key expires.

3. What happens with a poison entry

The third state is a wrong value that has entered the cache, from a bug, a partial write or a serialisation change, and the question is how it leaves. If the answer is "when the TTL expires", the poison lives for the full TTL, and if the TTL is long, that is a long incident with a known cause and no lever.

The review wants a purge path. A version in the key so a schema change invalidates everything at once, an explicit purge on the write path for the values that must not be stale, and a kill switch that drops the cache entirely without a deploy. The cache that cannot be purged without a release is a cache whose worst case duration is your release cycle, which is the wrong number.

The serialisation half matters too. A cached object whose shape changes between deploys will deserialise into garbage or throw, and the review should ask what happens to an old entry read by new code, because deploys are the most common source of poison.

4. What happens during a deploy

The fourth state is the overlap of old and new code both reading and writing the cache, and it combines the previous three. Old code writes entries new code misreads, new code's invalidations miss old code's keys, and the stampede fires as the new nodes start cold.

The review question is whether the key scheme and the entry schema are compatible across the deploy window, and whether the invalidation messages reach both generations. Versioned keys are the cheap answer, because they make the generations disjoint and let the old entries die of TTL while the new generation builds its own space.

The TTL question underneath all four

TTL is the only lever most caches have for correctness, and the review should make the author state what the TTL is protecting against. A TTL chosen for freshness is a freshness decision with an availability cost. A TTL chosen for safety against poison is an admission that there is no purge path. A TTL that is "long, to protect the backend" is a stampede and poison amplifier. The number is fine. The absence of a stated reason is the defect.

What I look for as proof

The strongest signal is a test for each state. Seed the cache and read, expire the hot key under concurrent readers and assert one computation, write a poison entry and assert the purge removes it, and read an old schema entry with new code and assert the fallback rather than a crash. Four tests, one per incident, and a cache that passes them has been reviewed for the conditions under which caches actually fail.

The isolation test habit from the cache that returned another customer's data is the fifth, for the key dimensions, and it belongs in the same suite.

The rule I keep

A cache is reviewed by walking its four states, cold, hot, poisoned and mid deploy, and by demanding the levers for each: capacity or warm up for cold, single flight for hot, purge and versioning for poison, and compatible keys across the deploy. The happy path is the least important code in the layer, and it is the only part the casual review ever reads.

The downstream symptom of getting the hot key wrong is the latency tail in p99 and why averages lie, because the stampede is precisely a tail event.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud