Message Queues

Implement asynchronous communication patterns for event-driven architectures, background job processing, and service decoupling.

When to Use This Skill

Use message queues when:

• Long-running operations block HTTP requests (report generation, video processing)
• Service decoupling required (microservices, event-driven architecture)
• Guaranteed delivery needed (payment processing, order fulfillment)
• Event streaming for analytics (log aggregation, metrics pipelines)
• Workflow orchestration for complex processes (multi-step sagas, human-in-the-loop)
• Background job processing (email sending, image resizing)

Broker Selection Decision Tree

Choose message broker based on primary need:

Event Streaming / Log Aggregation

→ Apache Kafka

• Throughput: 500K-1M msg/s
• Replay events (event sourcing)
• Exactly-once semantics
• Long-term retention
• Use: Analytics pipelines, CQRS, event sourcing

Simple Background Jobs

→ Task Queues

• Python → Celery + Redis
• TypeScript → BullMQ + Redis
• Go → Asynq + Redis
• Use: Email sending, report generation, webhooks

Complex Workflows / Sagas

→ Temporal

• Durable execution (survives restarts)
• Saga pattern support
• Human-in-the-loop workflows
• Use: Order processing, AI agent orchestration

Request-Reply / RPC Patterns

→ NATS

• Built-in request-reply
• Sub-millisecond latency
• Cloud-native, simple operations
• Use: Microservices RPC, IoT command/control

Complex Message Routing

→ RabbitMQ

• Exchanges (direct, topic, fanout, headers)
• Dead letter exchanges
• Message TTL, priorities
• Use: Multi-consumer patterns, pub/sub

Already Using Redis

→ Redis Streams

• No new infrastructure
• Simple consumer groups
• Moderate throughput (100K+ msg/s)
• Use: Notification queues, simple job queues

Performance Comparison