The problem
One event needs to reach many independent consumers without the producer knowing them all.
The idea
Publish events to a topic; any number of subscribers receive them independently.
How it works
Producers publish to a topic; the broker fans out to all subscriber groups, each tracking its own position so consumers progress independently. Log-based brokers (Kafka) keep an immutable, replayable log partitioned by key — ordering holds per partition, and a consumer group spreads partitions across its members for parallelism. Queue-based brokers (SQS) delete on ack and don't replay. Replaying (reset the offset) is the log's superpower: reprocess history after a bug or to seed a new consumer.
The tradeoff
Loose coupling and easy fan-out, but no single place sees the whole flow — end-to-end tracing and global ordering get harder. Ordering is only per-partition, so a key's events must share a partition or you lose their order. More partitions = more parallelism but more rebalancing churn; delivery is at-least-once, so consumers still need idempotency.
In the wild
A "user signed up" event consumed by email, CRM, analytics, and onboarding services.
Deep dive
Flow
- Producer publishes to a topic, keyed for partition/order.
- Broker appends to a partition; each group tracks its offset.
- Group members split partitions for parallel consumption.
- Reset offsets to replay history into a new or recovering consumer.
Watch for
- Ordering is per-partition only, never global.
- Adding consumers past the partition count gains no parallelism.
- At-least-once delivery means consumers must dedup.
Common trap
Distinguish a replayable log (Kafka) from a delete-on-ack queue (SQS), and state per-partition ordering.
Common questions
What problem does Pub/sub & fan-out solve?
One event needs to reach many independent consumers without the producer knowing them all.
How does Pub/sub & fan-out work?
Producers publish to a topic; the broker fans out to all subscriber groups, each tracking its own position so consumers progress independently. Log-based brokers (Kafka) keep an immutable, replayable log partitioned by key — ordering holds per partition, and a consumer group…
What are the tradeoffs of Pub/sub & fan-out?
Loose coupling and easy fan-out, but no single place sees the whole flow — end-to-end tracing and global ordering get harder. Ordering is only per-partition, so a key's events must share a partition or you lose their order. More partitions = more parallelism but more…
Where is Pub/sub & fan-out used in production?
A "user signed up" event consumed by email, CRM, analytics, and onboarding services.