Message Queues — Pub/Sub Architecture with RabbitMQ, Kafka, and SQS Explained

A message queue is a communication mechanism where producers send messages to a buffer (queue) and consumers retrieve them asynchronously, decoupling the sender from the receiver for resilient, scalable processing.

Why Message Queues Matter

Direct service-to-service communication creates tight coupling. If service A calls service B directly and B is slow or down, A fails too. Message queues break this dependency. When an e-commerce order is placed, the order service publishes an “order_placed” event to a queue. The inventory service, payment service, and notification service all consume independently. If inventory is down, orders are still accepted — the messages wait in the queue until inventory recovers. Kafka handles trillions of messages per day at LinkedIn, Netflix, and Uber.

Plain-Language Explanation

Imagine a busy restaurant. Without a queue, the waiter would need to personally hand each order to the chef and wait for the food before taking the next order — that’s synchronous processing, slow and fragile. In a real restaurant, the waiter writes an order on a ticket and puts it on a spindle. The chef grabs tickets when ready. Multiple chefs can grab tickets from the same spindle. If a chef steps away, tickets keep accumulating. Other staff (dessert, drinks) can also grab specific tickets.

The spindle is your message queue. The waiter is the producer (publishes messages). The chefs are consumers (process messages). The spindle doesn’t care who put the ticket there or who takes it — the same spindle works with multiple waiters and chefs.

graph LR
    P1[Producer: Order Service] --> Q[(Message Queue)]
    P2[Producer: User Service] --> Q
    Q --> C1[Consumer: Inventory]
    Q --> C2[Consumer: Payment]
    Q --> C3[Consumer: Notification]
    Q --> C4[Consumer: Analytics]
    style Q fill:#e67e22,color:#fff
    style P1 fill:#3498db,color:#fff
    style P2 fill:#3498db,color:#fff
    style C1 fill:#27ae60,color:#fff
    style C2 fill:#27ae60,color:#fff
    style C3 fill:#27ae60,color:#fff
    style C4 fill:#27ae60,color:#fff

Message Queue vs Event Stream

Message queues (RabbitMQ, SQS) deliver each message to one consumer. When the consumer acknowledges processing, the message is removed. Great for task distribution and work queues.

Event streams (Kafka, Kinesis) persist messages and allow multiple consumers to read the same message independently. Each consumer has its own offset. Messages are retained for a configurable period regardless of consumption.

RabbitMQ Example with Python

# producer.py
import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

channel.queue_declare(queue='order_events', durable=True)

message = '{"order_id": 1234, "user_id": 42, "amount": 99.99}'
channel.basic_publish(
    exchange='',
    routing_key='order_events',
    body=message,
    properties=pika.BasicProperties(delivery_mode=2)  # Persistent
)
print(f"Published: {message}")
connection.close()

# consumer.py
import pika

def callback(ch, method, properties, body):
    print(f"Processing order: {body.decode()}")
    # Simulate work
    import time; time.sleep(0.5)
    print("Order processed successfully")
    ch.basic_ack(delivery_tag=method.delivery_tag)

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

channel.queue_declare(queue='order_events', durable=True)
channel.basic_qos(prefetch_count=1)  # Don't send more than 1 at a time
channel.basic_consume(queue='order_events', on_message_callback=callback)

print("Waiting for messages. Ctrl+C to exit.")
channel.start_consuming()

Expected output from consumer:

Waiting for messages. Ctrl+C to exit.
Processing order: {"order_id": 1234, "user_id": 42, "amount": 99.99}
Order processed successfully

Use Cases

Decoupling: The order service doesn’t need to know about inventory, payment, or notification services. It just publishes events. New services can subscribe without modifying the producer.

Buffering: During a flash sale, thousands of orders per second arrive. The message queue buffers them. Consumers process at their own pace without the database being overwhelmed.

Async processing: Email sending, image resizing, PDF generation — tasks that don’t need immediate response. The API returns instantly and the task processes in the background.

Load leveling: If consumers can only handle 100 orders per second but traffic spikes to 500, the queue grows during the spike and drains afterward.

Dead Letter Queues

Messages that can’t be processed successfully (after N retries) are moved to a dead letter queue (DLQ) for inspection. This prevents poison messages from blocking the main queue.

# Dead letter queue configuration for RabbitMQ
channel.exchange_declare(exchange='main', exchange_type='direct')
channel.exchange_declare(exchange='dlx', exchange_type='direct')

channel.queue_declare(queue='main_queue', arguments={
    'x-dead-letter-exchange': 'dlx',
    'x-dead-letter-routing-key': 'dead',
    'x-message-ttl': 60000,  # 60 seconds
})

channel.queue_declare(queue='dead_queue')
channel.queue_bind(queue='dead_queue', exchange='dlx', routing_key='dead')

Common Mistakes

1. No message persistence: If the broker restarts, unprocessed messages disappear. Always enable persistence (delivery_mode=2 in RabbitMQ, acks=all in Kafka).

2. Infinite retries: A consumer that infinitely retries a bad message can cause a backlog. Use a retry limit (e.g., 3 attempts) then send to a dead letter queue.

3. Consumers too slow with no prefetch limit: Without prefetch_count=1, RabbitMQ sends all messages to a single consumer, overwhelming it. Limit prefetch to let consumers gradually take work.

4. Not idempotent: Queues typically guarantee at-least-once delivery. Your consumer must handle duplicate messages gracefully (use an idempotency key or dedup table).

5. Monitoring blind spots: Without queue depth monitoring, a growing backlog silently turns into hours-long delays. Alert on queue depth exceeding normal thresholds.

Practice Questions

1. What is the difference between RabbitMQ and Kafka? RabbitMQ is a message broker for task queues — messages are removed after acknowledgment. Kafka is a distributed event log — messages persist and can be replayed. RabbitMQ targets one consumer per message; Kafka supports multiple independent consumers.

2. What does at-least-once delivery mean? The broker guarantees a message is delivered at least once to the consumer. If the consumer fails to acknowledge, the message is redelivered. Consumers must handle duplicates.

3. How do dead letter queues work? When a message fails processing after N retries or exceeds TTL, it’s routed to a DLQ. This prevents bad messages from blocking the main queue while preserving them for debugging.

4. What is the advantage of async processing with queues? The API responds immediately without waiting for slow operations (email, PDF generation). Users get faster responses, and background tasks scale independently.

5. How do you secure a message queue in production? Use TLS for transport encryption, authentication (username/password or certificates), network isolation (VPC/firewall rules), and access control for exchanges/queues.

Mini Project

Build a simple task queue that resizes images asynchronously:

# task_queue.py
import pika, json, time
from PIL import Image

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='image_tasks', durable=True)

def resize_image(image_path: str, width: int, height: int):
    task = json.dumps({"path": image_path, "width": width, "height": height})
    channel.basic_publish(exchange='', routing_key='image_tasks', body=task,
                          properties=pika.BasicProperties(delivery_mode=2))
    print(f"Queued: {image_path} -> {width}x{height}")

# Queue a few tasks
resize_image("photos/vacation.jpg", 800, 600)
resize_image("photos/selfie.jpg", 400, 400)
resize_image("photos/panorama.jpg", 1920, 1080)

connection.close()

Expected output:

Queued: photos/vacation.jpg -> 800x600
Queued: photos/selfie.jpg -> 400x400
Queued: photos/panorama.jpg -> 1920x1080

Cross-References

• System Design Overview
• Microservices Patterns
• Event-Driven Architecture
• Load Balancing
• IoT Overview