Message Queues in System Design: From Basics to Real-World Use Cases
Jenish
Mar 5, 2026
6–8 min read
A typical message structure consists of two main parts:
Headers: These contain metadata about the message, such as unique identifier, timestamp, message type, and routing information.
Body: The body contains the actual message payload or content. It can be in any format, including text, binary data, or structured data like JSON.
:-)
Real Life Use Of Message Queue
E-Commerce : Once an order is received, put the order in OrderQueue so that the customer does not have to be blocked.
Payment Service: Listens to OrderQueue, processes payment.
Inventory Service: Listens to PaymentCompletedQueue, updates stock.
Email Service: Listens to OrderConfirmedQueue, sends confirmation email.
:-)
Components of a Message Queues System
Message Producer: Messages are created and sent to the message queue by the message producer. Any program or part of a system that produces data for sharing can be considered this.
Message Queue: Until the message consumers consume them, the messages are stored and managed by a data structure or service called the message queue. It serves as a mediator or buffer between consumers and producers.
Message Consumer: Messages in the message queue must be retrieved and processed by the message consumer. Messages from the queue can be read concurrently by several users.
Message Broker (Optional): In some message queue systems, a message broker acts as an intermediary between producers and consumers, providing additional functionality like message routing, filtering, and message transformation.
:-)
How Message Queues Work?
Below are some steps to understand how message queues work:
Step 1: Sending Messages: The message producer creates a message and sends it to the message queue. The message typically contains data or instructions that need to be processed or communicated.
Step 2: Queuing Messages: The message queue stores the message temporarily, making available for one or more consumers. Messages are typically stored in a first-in, first out (FIFO) order.
Step 3: Consuming Messages: Message consumers retrieve messages from the queue when they are ready to process them. They can do this at their own pace, which enables asynchronous communication.
Step 4: Acknowledgment (Optional): In some message queue systems, consumers can send acknowledgments back to the queue, indicating that they have successfully processed a message. This is essential for ensuring message delivery and preventing message loss.
For example:
A simple example of a message queue is an email inbox. When you send an email, it is placed in the recipient's inbox. The recipient can then read the email at their convenience. This email inbox acts as a buffer between the sender and the recipient, decoupling them from each other.
:-)
Why do we need Message Queues?
Message Queue are needed to address a number of challenges in distributed systems, including:
Asynchronous Communication: Applications can send and receive messages without waiting for a response due to message queues. Building scalable and dependable systems requires this.
Decoupling: Message queues decouple applications from each other, allowing them to be developed independently. This makes systems more flexible and easier to maintain.
Scalability: Message queues can be scaled to handle large volumes of messages by adding more servers. This makes them ideal for high-traffic applications.
Reliability: Message queues can be designed to be highly reliable, with features such as message persistence, retries, and dead letter queues. This ensures that messages are not lost even in the event of failures.
Workflow Management: Message queues can be used to implement complex workflows, such as order processing and payment processing. This can help improve the efficiency and accuracy of these processes.
:-)
Types of Message Queues
There are two main types of message queues in system design:
1. Point-to-Point Message Queues
Point-to-point message queues are the simplest type of message queue. When a producer sends a message to a point-to-point queue, the message is stored in the queue until a consumer retrieves it. Once the message is retrieved by a consumer, it is removed from the queue and cannot be processed by any other consumer.
Point-to-point message queues can be used to implement a variety of patterns such as:
Request-Response: A producer sends a request message to a queue, and a consumer retrieves the message and sends back a response messages.
Work Queue: Producers send work items to a queue, and consumers retrieve the work items and process them.
Guaranteed Delivery: Producers send messages to a queue, and consumers can be configured retry retrieving messages until they are successfully processed.
2. Publish-Subscribe Message Queues
Publish-Subscribe Message Queues are more complex than point-to-point message queues. When a producer publishes a message to publish/subscribe queue, the message is routed to all consumers that are subscribed to the queue. Consumers can subscribe to multiple queues, and they can also unsubscribe from queues at any time.
Publish-Subscribe Message Queues are often used to implement real-time streaming applications, such as social media and stock market tickers.
They can also be used to implement event-driven architecture, where components of a system communicate with each other by publishing and subscribing to events.
:-)
Message Routing
Message Routing involves determining how messages are directed to their intended recipients. The following methods can be employed:
Topic-Based Routing: Messages are sent to topics or channels, and subscribers express interest in specific topics. Messages are delivered to all subscribers of a particular topic.
Direct Routing: Messages are sent directly to specific queues or consumers based on their addresses or routing keys.
Content-Based Routing: The routing decision is based on the content of the message. Filters or rules are defined to route messages that meet specific criteria.
:-)
Scalability of Message Queues
Scalability is essential to ensure that a message queue system can handle increased loads efficiently. To achieve scalabilityrelia:
Distributed Queues: Implement the message queue as a distributed system with multiple nodes, enabling horizontal scaling.
Partitioning: Split queues into partitions to distribute message processing across different nodes or clusters.
Load Balancing: Use load balancers to evenly distribute incoming messages to queue consumers.
:-)
Dead Letter Queues and Message Prioritization in Message Queues
1. Dead Letter Queues
Dead Letter Queues (DLQs) are a mechanism for handling messages that cannot be processed successfully. This includes:
Messages with errors in their content or format.
Messages that exceed their time-to-live (TTL) or delivery attempts.
Messages that cannot be delivered to any consumer.
DLQs provide way to investigate and potentially reprocess failed messages while preventing them from blocking the system.
2. Message Prioritization
Message Prioritization is the process of assigning priority levels to messages to control their processing order. Prioritization criteria can include:
Urgency: Messages with higher priority may need to processed before lower-priority messages.
Message Content: Messages containing critical information or commands may receive higher priority.
Business Rules: Custom business rules or algorithms may be used to determine message priority.
:-)
Message Queue Implementation
Message Queues can be implemented in a variety of ways, but they typically follow a simple pattern:
Producer: An application that sends messages to a queue.
Message Broker: A server that stores and forwards messages between producers and consumers.
Consumer: An application that receives messages from a queue.
Problem Statement:
In a real-world scenario, you might want to consider using a dedicated message queue service like RabbitMQ or Apace Kafka for distributed systems.