Day 08 · System Design
Message Queues vs. Event Streams (RabbitMQ vs. Apache Kafka)
When to use a queue that deletes messages after delivery, and when to use a replayable, partitioned event log.
01Overview & core concept
Asynchronous messaging decouples microservices, absorbs traffic spikes (load leveling), and enables reliable background processing. Architectures primarily fall into two paradigms.
Message Queues (Smart Broker, Dumb Consumer)
Messages are delivered to individual consumers and deleted from the queue once acknowledged (e.g., RabbitMQ, AWS SQS).
Event Streams / Distributed Commit Logs (Dumb Broker, Smart Consumer)
Messages are appended to an immutable, partitioned, ordered log. Consumers maintain their own read offsets and pull data at their own pace without deleting messages (e.g., Apache Kafka, Apache Pulsar).
02Architecture & request flow
MESSAGE QUEUE (e.g., RabbitMQ)
┌──────────┐ ┌───────────────────────────────────┐ ┌────────────┐
│ Producer ├───►│ Exchange ──► [ Queue ] (Msg Acked │───►│ Consumer A │
└──────────┘ │ & Deleted) │ └────────────┘
└───────────────────────────────────┘
EVENT STREAM / LOG (e.g., Apache Kafka)
┌──────────┐ ┌───────────────────────────────────┐ ┌────────────────┐
│ Producer ├───►│ Partition 0: [0][1][2][3][4][5]... │───►│ Consumer Grp 1 │ (Offset: 5)
└──────────┘ │ │ └────────────────┘
│ Partition 1: [0][1][2][3]... │───►│ Consumer Grp 2 │ (Offset: 2)
└───────────────────────────────────┘ └────────────────┘
(Immutable Log: Retained by TTL)Core differences
1. Message lifecycle
Queue: ephemeral, acknowledged messages are discarded immediately by the broker. Stream: persistent, messages are retained on disk for a configurable window (e.g., 7 days) and can be replayed from any historical offset.
2. Delivery & polling model
Queue (push): the broker pushes messages to connected workers based on prefetch limits. Stream (pull): consumers poll partitions in batches, giving built-in backpressure.
3. Ordering guarantees
Queue: FIFO ordering is lost when multiple competing consumers process messages concurrently from one queue. Stream: strict ordering is preserved within each partition, based on the message key (e.g., user_id).
Delivery semantics
At-Most-Once
Message sent without retries (can be lost, never duplicated).
At-Least-Once (standard)
Retries on failure until acknowledgment. Requires idempotent consumers to prevent duplicate execution.
Exactly-Once Processing
Supported via transactional producers and coordinated consumer commits (e.g., Kafka Transactions / Flink state).
03Real-world example: Uber Real-Time Event Architecture
Kafka for Core Telemetry
Uber produces trillions of events daily (driver GPS pings, rider app events, marketplace pricing signals) to Apache Kafka clusters. Multiple independent systems (surge pricing, fraud detection, real-time analytics, data lake ingestion) read from the same partitioned streams without impacting one another.
Task Queues (RabbitMQ / SQS) for Discrete Work
Discrete transactional workflows, such as generating a PDF receipt, charging a credit card, or sending an SMS, are delegated to task queues with explicit Dead Letter Queues (DLQs) and individual message acknowledgments.
04Architectural trade-offs
| Dimension | Message Queues (RabbitMQ) | Event Streams (Kafka) |
|---|---|---|
| Throughput | 10k–50k msgs/sec per node (CPU/memory bound). | 1M+ msgs/sec per cluster (sequential disk I/O, zero-copy sendfile). |
| Message Routing | Highly flexible (Topic, Direct, Fanout, Headers exchanges). | Simple key-based partition hashing (hash(key) % partitions). |
| Replayability | None (once consumed, data is gone). | Full historical replay from any timestamp or offset. |
| Consumer Scaling | Any number of workers can pull from a single queue. | Max parallel consumers in a consumer group = number of partitions. |
| Operational Overhead | Low to moderate. | Moderate to high (cluster rebalancing, ZooKeeper/KRaft quorum management). |
Failure modes & mitigations
Poison Pill Messages
A malformed payload crashes the consumer upon reading. Mitigation: catch parsing exceptions, route invalid messages to a Dead Letter Queue (DLQ) for manual inspection, and commit the offset so the pipeline does not stall.
Consumer Lag
Processing speed falls behind the ingestion rate in Kafka. Mitigation: scale up partition count and consumer instances within the consumer group, or optimize consumer batch processing.
05Interview question & approach
Design a high-throughput notification service that handles 100,000 notifications/second, delivers critical alerts (like OTP verification codes) in under 2 seconds, and handles downstream SMS/Email provider outages without dropping messages.
Queue Segregation & Priority Tiers
Do not mix low-priority bulk marketing emails with high-priority transactional OTPs in the same queue. Create isolated queues/topics: notifications.critical.otp, notifications.transactional, notifications.marketing.bulk.
Ingestion & Buffering Layer
Use an event log (Kafka) or distributed queue (RabbitMQ/SQS) with autoscaled worker consumer pools.
Handling Vendor Outages & Rate Limits
When third-party vendors (e.g., Twilio, SendGrid) return 429 Rate Limited or 500 Server Error, use exponential backoff with jitter. Forward persistently failed messages to a retry queue with delayed delivery or a DLQ.
Idempotency & Deduplication
Assign a unique idempotency_key (e.g., UUID(order_id + event_type)) to every notification. Consumers check and set an atomic lock or cache key in Redis before dispatching, so no customer receives duplicates during retries.