Cache stampede & hot keys

The problem

A popular cached key expires and suddenly thousands of requests all miss and hammer the DB at once.

The idea

Prevent the herd: stop everyone from rebuilding the same key simultaneously.

How it works

Three defenses stack. (1) Single-flight / request coalescing — one rebuild per key, the rest wait on its result, so N concurrent misses collapse to 1 DB read. (2) Probabilistic early expiration (XFetch) — each reader rolls a dice weighted by recompute cost and remaining TTL, so one lucky request refreshes the key just before it expires while everyone else still serves the cached value, removing the synchronized cliff. (3) Stale-while-revalidate — serve the old value and refresh in the background. Add TTL jitter (±10–30%) so keys written together don't expire in the same instant.

The tradeoff

A lock/single-flight adds a few ms to the one rebuild and needs a lock that releases on crash — a stuck lock is its own outage. And stampede control does nothing for a single hot key (a viral post) that saturates the one node owning it; that's a distribution problem you fix by replicating the key across nodes or fronting it with a per-process local cache, trading a little staleness for spread.

In the wild

A classic outage shape — a homepage cache expiring under peak traffic, or one celebrity key melting a shard.

Interview deep dive

Flow

1. A hot key nears expiry under heavy read traffic.
2. One request wins the single-flight lock or XFetch dice and recomputes.
3. Other readers wait briefly or serve the slightly-stale value.
4. Cache refills with a jittered TTL; the herd never hits the DB.

Watch for

• A lock without a timeout deadlocks if the holder crashes mid-rebuild.
• Single-flight coalesces per node — 50 servers can still fire 50 rebuilds.
• Hot-key saturation is a distribution problem, separate from the expiry herd.

Interviewer trap

Separate the expiry herd (single-flight + jitter) from the single hot key (replication / local cache).

Related Academy