Message Queues: When and How to Use Them

One of the most consequential architectural decisions is whether to process a request synchronously or pass it through a queue. Get it right and your system scales gracefully.

1. Sync vs Async Decision Tree

Can the user wait for the response?
├── Yes (< 500ms) → Process synchronously (HTTP)
├── Not really (500ms-5s) → WebSocket/SSE
└── No (> 5s) → Queue it

Good candidates for queuing: Email sending, image processing, data export, webhook delivery, index updates, batch processing.

Bad candidates: User login, payment authorization, read-after-write critical path.

2. RabbitMQ vs Kafka

Criteria	RabbitMQ	Kafka
Primary model	Message broker (push)	Log-based (pull)
Message routing	Complex (exchanges, bindings)	Simple (topics)
Throughput	10K-50K msg/s	100K-1M+ msg/s
Message retention	Deleted after ack	Configurable retention
Latency	Sub-millisecond	Few milliseconds

Choose RabbitMQ when: You need complex routing, sub-ms latency, per-message acknowledgments with DLQs.

Choose Kafka when: You need to replay messages, throughput > 50K msg/s, persistent event log, multiple consumer groups.

3. RabbitMQ Patterns

Direct Exchange (point-to-point):

// Producer
channel.assertExchange('orders', 'direct', { durable: true });
channel.publish('orders', 'order.created', Buffer.from(JSON.stringify(order)), {
  persistent: true,
});

// Consumer
channel.assertExchange('orders', 'direct', { durable: true });
const queue = await channel.assertQueue('email-service', { durable: true });
channel.bindQueue(queue.queue, 'orders', 'order.created');
channel.consume(queue.queue, (msg) => {
  const order = JSON.parse(msg.content.toString());
  channel.ack(msg);
});

Topic Exchange (pub/sub):

channel.assertExchange('events', 'topic', { durable: true });
channel.publish('events', 'user.created.eu', Buffer.from(JSON.stringify(data)));

// Consumer: all user events
channel.bindQueue(queue.queue, 'events', 'user.*');

Dead Letter Queue:

channel.assertQueue('orders.retry', {
  durable: true,
  arguments: {
    'x-dead-letter-exchange': 'orders.dlx',
    'x-message-ttl': 30000,
  },
});

4. Kafka Patterns

Producer:

const { Kafka } = require('kafkajs');
const kafka = new Kafka({ clientId: 'order-service', brokers: ['kafka-1:9092'] });
const producer = kafka.producer();
await producer.connect();

await producer.send({
  topic: 'order-events',
  messages: [{ key: order.id, value: JSON.stringify(order) }],
});

Consumer Group:

const consumer = kafka.consumer({ groupId: 'email-service' });
await consumer.connect();
await consumer.subscribe({ topic: 'order-events', fromBeginning: false });

await consumer.run({
  eachMessage: async ({ message }) => {
    const order = JSON.parse(message.value.toString());
    await sendEmail(order);
  },
});

5. Common Anti-Patterns

1. Queue for request-response: If you need a synchronous response, use HTTP directly.

2. Queue proliferation: One exchange with routing keys (RabbitMQ) or fewer topics with partition keys (Kafka).

3. Large messages: Store data externally, pass a reference:

// Wrong
producer.send({ topic: 'thumbnails', messages: [{ value: hugeImage }] });

// Right
producer.send({ topic: 'thumbnails', messages: [{ value: JSON.stringify({ s3Key: '...' }) }] });

4. No monitoring:

# RabbitMQ
rabbitmqctl list_queues name messages consumers

# Kafka
kafka-consumer-groups --bootstrap-server localhost:9092 \
  --group my-group --describe  # Look at LAG

Summary

Use Case	Recommended
Task queues (email, notifications)	RabbitMQ
Event sourcing / CDC	Kafka
Log aggregation	Kafka
Pub/sub with routing	RabbitMQ
Simple background jobs	RabbitMQ or Redis (Bull)

Choose the tool that matches your use case, not the one most popular on Hacker News.