Unique ID Generator

Requirements

The approach

Snowflake: [timestamp | machine_id | sequence]. Each host generates locally with no coordination; machine_id assigned at startup; sequence handles same-ms bursts.

Key components

Per-host generator · ZooKeeper/etcd for machine_id assignment

Numbers that matter

• Snowflake encodes 41 bits of millisecond timestamp — enough for ~69 years from a custom epoch before overflow (Twitter's epoch is Nov 4, 2010)
• 10 bits of machine ID supports up to 1024 worker nodes; 12 bits of sequence allows 4096 IDs per millisecond per node, giving ~4M IDs/sec per node
• A single auto-increment MySQL sequence tops out at roughly ~50K inserts/sec under write pressure — 80× less throughput than a Snowflake node with zero coordination
• UUID v4 is 128 bits (vs Snowflake's 64 bits), provides no ordering, and consumes twice the storage in indexes — index fragmentation on UUID primary keys degrades B-tree performance significantly at hundreds of millions of rows

Senior deep-dive

The core insight is that coordination is the enemy of throughput — Snowflake-style IDs move all state to the local host clock and a sequence counter, generating globally unique IDs with zero network calls.

Machine ID assignment is the coordination point you can't eliminate — use ZooKeeper/etcd ephemeral nodes or a pre-allocated range from a central service, and guard against clock skew with a startup check.

Monotonicity is not guaranteed across machines — IDs from the same timestamp window on different hosts are unordered; if you need global ordering, you need a global sequencer (and its throughput ceiling).

Machine ID assignment: the only coordination you need

The entire point of Snowflake is one-time coordination at startup rather than per-ID. Workers claim a machine ID via an ephemeral ZooKeeper/etcd node (auto-released on crash) or a centralized allocator that hands out IDs from a pre-divided space. The second approach (range allocation, e.g. 'worker gets IDs 100-199') works even if the allocator goes down — workers keep using their range until restart. Never embed machine ID in config — ops will duplicate it across hosts and create silent ID collisions.

Clock skew: the silent killer

NTP can move a clock backward by tens to hundreds of milliseconds. If a Snowflake worker generates ID at `t=1000`, then NTP slews the clock to `t=950`, the next ID has a lower timestamp than the previous one — violating monotonicity and potentially duplicating sequence+timestamp combos. The fix: detect backward clock movement, block ID generation, and log an alert. For large backward jumps (>a few seconds), the worker should exit and page an engineer rather than waiting silently.

Sequence exhaustion: the burst problem

12 bits of sequence gives 4096 IDs per millisecond per worker. At extremely high write rates (bulk inserts, event storms), you can exhaust the sequence within one millisecond — the worker must then either spin-wait for the next millisecond (adds ~1ms of latency at worst) or pre-buffer IDs by advancing the logical clock. Pre-buffering breaks monotonicity guarantees across the buffer boundary. Most systems choose to spin-wait, since sequence exhaustion should be rare and the wait is bounded.

Bit layout is a design decision, not a given

Twitter's layout [41 timestamp | 10 machine | 12 sequence] is not universal. Instagram split machine bits into datacenter + worker ID. If you need more workers (e.g. 10K), steal bits from sequence (reduces IDs/ms/node). If you need longer timestamps (beyond 69 years), steal bits from machine or sequence. The key constraint: total must fit in 63 signed bits (Java/most languages treat int64 as signed). Custom epochs buy you back decades — don't use Unix epoch as your zero point.

Sortable IDs have database implications

Time-ordered IDs make B-tree index insertions sequential — new rows always go to the rightmost leaf page, eliminating random page writes. This is a massive write performance win over UUIDs (which scatter inserts across the entire index). The tradeoff: sortable IDs leak business intelligence (volume, velocity) to anyone who compares IDs over time. Some systems XOR the timestamp bits or use a Feistel cipher to scramble ordering while preserving uniqueness.

What breaks at scale

Worker ID exhaustion hits when you exceed 1024 concurrent workers — you either need more bits (redesign the scheme) or a two-level hierarchy (datacenter × worker). Decommissioned workers whose IDs are recycled too quickly risk ID reuse if any downstream system cached old IDs; keep a tombstone TTL (at least one max-cache-lifetime) before reusing machine IDs. The final failure mode: epoch rollover — at 69 years, the timestamp overflows; build a monitoring alert for the 5-year warning mark, because the system will silently start generating duplicate IDs.

In production

Twitter's original Snowflake service (open-sourced 2010) inspired nearly every high-scale ID generator. Instagram's ID scheme used PostgreSQL schemas with a custom function: `shard_id + timestamp_ms + sequence` packed into 64 bits — same idea but executed inside Postgres to avoid a separate service. Discord uses Snowflakes with a custom epoch. The real engineering challenge is clock synchronization: NTP slew can move a system clock backward, which would generate duplicate IDs; Snowflake workers must refuse to generate IDs during clock-backward periods (wait until the clock catches up), and this pause is detectable as a latency spike.

Common mistakes

• Using a central DB sequence (bottleneck/SPOF)
• Ignoring clock drift
• UUIDv4 as a primary key on huge B-trees (random inserts)

Related System Design Library