Pub/sub & fan-out

The problem

One event needs to reach many independent consumers without the producer knowing them all.

The idea

Publish events to a topic; any number of subscribers receive them independently.

How it works

Producers publish to a topic; the broker fans out to all subscriber groups, each tracking its own position so consumers progress independently. Log-based brokers (Kafka) keep an immutable, replayable log partitioned by key — ordering holds per partition, and a consumer group spreads partitions across its members for parallelism. Queue-based brokers (SQS) delete on ack and don't replay. Replaying (reset the offset) is the log's superpower: reprocess history after a bug or to seed a new consumer.

The tradeoff

Loose coupling and easy fan-out, but no single place sees the whole flow — end-to-end tracing and global ordering get harder. Ordering is only per-partition, so a key's events must share a partition or you lose their order. More partitions = more parallelism but more rebalancing churn; delivery is at-least-once, so consumers still need idempotency.

In the wild

A "user signed up" event consumed by email, CRM, analytics, and onboarding services.

Interview deep dive

Flow

1. Producer publishes to a topic, keyed for partition/order.
2. Broker appends to a partition; each group tracks its offset.
3. Group members split partitions for parallel consumption.
4. Reset offsets to replay history into a new or recovering consumer.

Watch for

• Ordering is per-partition only, never global.
• Adding consumers past the partition count gains no parallelism.
• At-least-once delivery means consumers must dedup.

Interviewer trap

Distinguish a replayable log (Kafka) from a delete-on-ack queue (SQS), and state per-partition ordering.

Related Academy