Distributed Unique ID Generation (Twitter Snowflake & UUIDv7)

01Overview & core concept

In a single relational database, unique primary keys are easily generated using AUTO_INCREMENT or sequences. In a distributed, sharded architecture with hundreds of microservices writing concurrently, relying on a centralized database for ID generation creates a single point of failure (SPOF) and a severe throughput bottleneck. Distributed systems need IDs that satisfy four constraints.

• Global Uniqueness

  Zero collisions across all servers and regions.

• Rough Time-Sortability (k-sortable)

  IDs increase chronologically to maintain index clustering in B-trees, preventing random disk I/O and page fragmentation.

• High Throughput & Low Latency

  Generating an ID must be an in-memory operation (under 1 microsecond) with zero network hops per request.

• Compact Size

  Typically 64 bits (fits a standard unsigned BIGINT) or 128 bits.

02Architecture & request flow

 0        1                                42              52            63
┌────────┬────────────────────────────────┬───────────────┬─────────────┐
│ 1 Bit  │       41 Bits Timestamp        │ 10 Bits Node  │ 12 Bits Seq │
│ Unused │ (Milliseconds since Epoch)     │ (Machine ID)  │  (Counter)  │
└────────┴────────────────────────────────┴───────────────┴─────────────┘

The Twitter Snowflake algorithm generates 64-bit, roughly time-ordered IDs without inter-node network coordination.

03Real-world example: Discord Message & Channel IDs

• Snowflake IDs Across All Entities

  Discord uses 64-bit Snowflake IDs for messages, guilds, channels, and users.

• Efficient Pagination

  Because Snowflake IDs are chronological, Discord paginates message queries with simple inequalities instead of heavy SQL offsets:

  SELECT * FROM messages 
  WHERE channel_id = :channel_id AND message_id < :oldest_seen_snowflake 
  ORDER BY message_id DESC 
  LIMIT 50;

• Worker ID Allocation

  Worker IDs are assigned automatically on startup using ZooKeeper, etcd, or Kubernetes StatefulSet ordinal indices (app-0, app-1, etc.).

04Architectural trade-offs

Failure modes & mitigations

• Clock Drift & Backward Clock Shifts (NTP Sync)

  If a server's clock adjusts backward (e.g., via NTP slew or step), generating an ID with a past millisecond can collide with earlier IDs. Mitigations: if the shift is under about 5 ms, pause and spin-wait until the clock catches up; if it is significant, refuse ID generation and trigger an alert, or borrow sequence bits to advance the virtual timestamp.

• Sequence Overflow in 1 Millisecond

  If a single worker generates more than 4,096 IDs within 1 ms, it must block or spin until the clock advances to the next millisecond.