The problem
Producers can outpace consumers; the queue grows without bound and everything falls over.
The idea
Apply backpressure to slow producers, and route poison messages to a dead-letter queue.
How it works
Producers can outpace consumers, so an unbounded queue just trades a crash for a slow, memory-eating death — a growing backlog is latency debt, every item waits longer. Bound the queue and apply backpressure: block/slow producers, or shed low-priority load and fail fast. Messages that fail repeatedly go to a dead-letter queue with a redrive path, so one poison message can't block the partition forever. Consumer lag is the key signal — it shows the backlog growing before users feel it.
The tradeoff
Backpressure protects the system but means rejecting or delaying work — you choose which load to shed. An unbounded queue hides the problem until OOM; a bounded one surfaces it as visible rejection you can react to. DLQs prevent head-of-line blocking but silently swallow failures unless you alert on DLQ depth and actually redrive — an unwatched DLQ is just a data-loss bucket.
In the wild
Kafka consumer lag dashboards + DLQ topics are standard ops hygiene.
Deep dive
Flow
- Bound the queue so backlog can't grow without limit.
- On saturation, apply backpressure or shed low-priority work.
- Route repeatedly-failing messages to a DLQ.
- Alert on consumer lag and DLQ depth; redrive after a fix.
Watch for
- Unbounded queues convert overload into an OOM crash.
- A rising backlog is latency debt — every item waits longer.
- An unmonitored DLQ silently loses data.
Common trap
Treat consumer lag as a product SLI and name the DLQ + redrive + alert loop.
Common questions
What problem does Backpressure & DLQs solve?
Producers can outpace consumers; the queue grows without bound and everything falls over.
How does Backpressure & DLQs work?
Producers can outpace consumers, so an unbounded queue just trades a crash for a slow, memory-eating death — a growing backlog is latency debt, every item waits longer. Bound the queue and apply backpressure: block/slow producers, or shed low-priority load and fail fast.…
What are the tradeoffs of Backpressure & DLQs?
Backpressure protects the system but means rejecting or delaying work — you choose which load to shed. An unbounded queue hides the problem until OOM; a bounded one surfaces it as visible rejection you can react to. DLQs prevent head-of-line blocking but silently swallow…
Where is Backpressure & DLQs used in production?
Kafka consumer lag dashboards + DLQ topics are standard ops hygiene.