What Is Message Queue?
A message queue is middleware that enables asynchronous communication between services. Producers send messages to a queue; consumers process them independently. This decouples systems, handles traffic spikes, and ensures messages aren't lost. Popular options: RabbitMQ, Amazon SQS, Apache Kafka.
How Message Queue Works
When a user uploads a video, don't process it in the request. Send a message to a queue: {userId, videoUrl, format}. A worker service picks up the message, transcodes the video, and updates the status. The user gets an instant response while processing happens in the background.
Kafka handles event streaming at massive scale — LinkedIn processes 7 trillion messages/day through Kafka. RabbitMQ is simpler for task queues. Amazon SQS is fully managed with zero infrastructure.
Why Developers Use Message Queue
Message queues are essential for microservices, background job processing, and handling traffic spikes. If your web request takes longer than 500ms, consider moving the slow part to a queue.
Key Concepts
- Producer/Consumer — Producers add messages to the queue; consumers pull and process them independently
- At-Least-Once Delivery — Guarantees every message is processed at least once — consumers must handle duplicates
- Dead Letter Queue — Messages that fail processing repeatedly are moved here for investigation
- Backpressure — When consumers can't keep up, messages queue up — the queue absorbs traffic spikes
Frequently Asked Questions
Kafka vs RabbitMQ?
Kafka for high-throughput event streaming and log aggregation. RabbitMQ for traditional task queues and request-reply patterns. Kafka retains messages; RabbitMQ deletes after consumption.
When do I need a message queue?
When operations are slow (video processing, email sending), when services need decoupling, or when you need to handle traffic spikes without dropping requests.