Your Cache Is Serving Stale Data and the Bug Is in the Invalidation

The short answer

Most stale cache bugs come from one of four things:

1. A race between write and invalidate, where a concurrent read repopulates the old value
2. Invalidating one key when several derived keys hold the same data
3. Multiple writers, where one path updates the database without invalidating
4. A cache that fails open on delete, so an invalidation error is silently ignored

The safest simple pattern is write to the database, then delete the cache key, not update it. And always set a TTL, even when you invalidate correctly, because it bounds how long a missed invalidation can hurt you.

Why delete rather than update

The instinct is to keep the cache fresh:

await db.users.update(id, data);
await cache.set(`user:${id}`, data);      // looks efficient

The problem is that two concurrent writes can land in the cache in the opposite order to the database:

t=0   A writes DB (name=Alice)
t=1   B writes DB (name Bob)
t=2   B writes cache (Bob)
t=3   A writes cache (Alice)      <- cache now disagrees with the database

The database has Bob. The cache has Alice, indefinitely, until the TTL expires.

Deleting instead:

await db.users.update(id, data);
await cache.del(`user:${id}`);

Both writers delete. The next read repopulates from the database, which is authoritative. Slightly less efficient, substantially harder to get wrong.

The read repopulation race

Deleting is safer and not free of races.

t=0   Reader gets a cache miss for user:1
t=1   Reader queries DB, gets name=Alice
t=2   Writer updates DB to Bob
t=3   Writer deletes cache key
t=4   Reader writes Alice into the cache      <- stale, and no TTL until it expires

The reader's write lands after the invalidation, resurrecting the old value.

This is narrow and it happens, particularly when the read path is slow. Three mitigations:

A TTL bounds the damage. Always set one. Even five minutes converts a permanent inconsistency into a temporary one, and this is the single highest value habit in caching.

Delete twice, with a delay. Delete, write to the database, then delete again after a short delay covering the read window:

await cache.del(key);
await db.users.update(id, data);
setTimeout(() => cache.del(key).catch(() => {}), 500);

Inelegant, effective, and widely used.

Set with NX on the read path, so a populating write does not overwrite an existing value:

await cache.set(key, value, { EX: 300, NX: true });

Does not fully close it and it narrows the window considerably.

Multiple keys holding the same data

The one people most often miss.

cache.set(`user:${id}`, user);
cache.set(`user:email:${email}`, user);
cache.set(`team:${teamId}:members`, members);   // contains this user
cache.set(`search:engineers`, results);          // contains this user

Updating the user and deleting only user:${id} leaves three stale copies.

Two approaches.

Tag based invalidation. Track which keys contain a given entity:

async function cacheWithTags(key, value, tags, ttl) {
  const tx = redis.multi();
  tx.set(key, JSON.stringify(value), { EX: ttl });
  for (const tag of tags) {
    tx.sAdd(`tag:${tag}`, key);
    tx.expire(`tag:${tag}`, ttl + 60);
  }
  await tx.exec();
}

async function invalidateTag(tag) {
  const keys = await redis.sMembers(`tag:${tag}`);
  if (keys.length) await redis.del(keys);
  await redis.del(`tag:${tag}`);
}

Versioned keys, which avoids deletion entirely:

const version = await redis.get(`ver:user:${id}`) ?? "0";
const key = `user:${id}:v${version}`;

// to invalidate, bump the version
await redis.incr(`ver:user:${id}`);

Old keys become unreachable and expire on their own. Elegant, and it costs an extra read per lookup, so it suits things read far more often than written.

Multiple writers

A background job, an admin tool, a database migration, or a direct SQL fix that updates rows without going through the code path that invalidates.

This is an architecture problem rather than a caching one. Two ways to address it:

Funnel all writes through one layer that owns invalidation. A repository or service layer that no other code bypasses.

Invalidate from the database. Change data capture, or Postgres LISTEN and NOTIFY on a trigger, so any write invalidates regardless of which code caused it:

CREATE FUNCTION notify_user_change() RETURNS trigger AS $$
BEGIN
  PERFORM pg_notify('cache_invalidate', 'user:' || NEW.id);
  RETURN NEW;
END $$ LANGUAGE plpgsql;

CREATE TRIGGER users_cache_invalidate
AFTER INSERT OR UPDATE OR DELETE ON users
FOR EACH ROW EXECUTE FUNCTION notify_user_change();

The database becomes the source of invalidation events, which is the only place that sees every write. More machinery, and it is the version that is actually correct.

Failing silently on delete

await db.users.update(id, data);
try {
  await cache.del(key);
} catch (err) {
  logger.warn({ err }, "cache invalidation failed");   // and then?
}

If the delete fails and you continue, the cache is stale for the full TTL and nothing will tell you.

Decide deliberately. Either the invalidation is best effort, in which case a short TTL is doing the real work and you should say so, or it is required, in which case the write should fail or be retried.

The outbox pattern applies here as it does to webhook side effects: record the invalidation intent in the same transaction as the write, and let a worker perform it with retries.

Also worth having a metric on invalidation failures. A rising failure rate is the leading indicator of widespread staleness and it usually generates no other signal.

Thundering herd on expiry

A popular key expires and a thousand concurrent requests all miss, all query the database, and all repopulate.

Probabilistic early expiration refreshes slightly before the TTL, spread randomly:

const { value, expiresAt } = await getWithMeta(key);
const remaining = expiresAt - Date.now();
if (remaining < 60_000 && Math.random() < 0.1) {
  refreshInBackground(key);
}
return value;

A lock on repopulation, so one request queries and the rest wait or serve stale:

const gotLock = await redis.set(`lock:${key}`, "1", { NX: true, EX: 10 });
if (gotLock) {
  const fresh = await loadFromDb();
  await cache.set(key, fresh, { EX: 300 });
  return fresh;
}
return staleValue ?? await loadFromDb();

Jitter the TTLs so keys populated together do not expire together:

const ttl = 300 + Math.floor(Math.random() * 60);

That last one is one line and it prevents a synchronised expiry storm, which is the same reasoning as jitter in retry backoff.

Debugging a staleness report

Compare directly. Read the cache and the database for the same key and diff them. Obvious, and it distinguishes "the cache is stale" from "the write never happened", which is a completely different investigation.

Check the TTL:

redis-cli TTL user:8823

-1 means no expiry, which is almost always a bug. -2 means the key does not exist, so your staleness is somewhere else.

Log cache operations with the request id so you can reconstruct the sequence for one user:

logger.debug({ event: "cache.miss", key, requestId });
logger.debug({ event: "cache.invalidate", key, requestId, reason: "user.update" });

Reading the interleaving of hits, misses, sets, and deletes across concurrent requests is how you find a race. It is the same reason logging beats stepping for ordering problems: you need the history, not a snapshot.

Watch keys in real time during a reproduction:

redis-cli --scan --pattern 'user:8823*'
redis-cli MONITOR | grep 8823        # development only, very heavy

The defaults I would set

• TTL on every key, always, with jitter
• Delete rather than update on write
• One layer owning writes and invalidation
• Metrics on hit rate, miss rate, and invalidation failures
• A documented answer to "what happens if invalidation fails"

Most caching bugs I have chased came from a missing TTL plus an invalidation path that had one gap in it. Those two defaults alone would have bounded almost all of them to a few minutes of staleness rather than indefinite.