Messaging is the backbone of distributed systems. It enables services to communicate asynchronously, decouple dependencies, and build resilient architectures. This article covers the two most fundamental messaging patterns: publish-subscribe and request-reply.

Messaging Patterns: Pub/Sub and Request/Reply

The Case for Messaging

Direct HTTP calls between services create tight coupling. When Service A calls Service B directly:

  • Service A must know Service B's location and API contract.

  • Service A is blocked while Service B processes the request.

  • If Service B is down, Service A fails.

  • Scaling Service B does not help Service A's performance.

Messaging intermediaries (message brokers like Kafka, RabbitMQ, SQS, or Pub/Sub) solve these problems by decoupling senders from receivers.

Publish-Subscribe (Pub/Sub)

In the pub/sub pattern, publishers send messages to a topic without knowing who the subscribers are. Subscribers receive messages from topics they have subscribed to.

[Publisher] -> [Topic] -> [Subscriber A]

-> [Subscriber B]

-> [Subscriber C]

How Pub/Sub Works

  • Topics are named channels that hold messages.

2\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\. Publishers send messages to a topic. 3\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\. Subscribers register interest in a topic and receive all messages published to it. 4\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\. Each subscriber receives a copy of every message (fan-out delivery).

Use Cases

Event notification. When a user signs up, publish a UserRegistered event. Multiple subscribers react: the email service sends a welcome email, the analytics service records the event, the CRM service creates a contact.

Broadcasting. A live sports app publishes score updates. Thousands of clients receive updates in real-time.

Log aggregation. Multiple services publish log entries to a central topic. A log processing service stores and indexes them.

Example with Kafka

Publisher

producer.send('order-events', {

'type': 'OrderPlaced',

'order_id': '123',

'customer_id': '456',

'total': 99.99

})

Subscriber

@kafka_listener('order-events')

def handle_order_placed(event):

if event['type'] == 'OrderPlaced':

inventory_service.reserve_inventory(event['order_id'])

notification_service.send_confirmation(event['customer_id'])

Pub/Sub is ideal for one-to-many communication where the publisher does not need a response.

Request-Reply

The request-reply pattern is fundamentally different from pub/sub. A sender sends a request and expects a response. The two are correlated so the sender knows which response goes with which request.

In messaging systems, request-reply requires correlation:

[Requestor] -> [Request Queue] -> [Replier]

[Requestor] <\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- [Reply Queue] <\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\- [Replier]

Correlation ID

The key mechanism is the correlation ID. The requestor includes a unique ID in the request message. The replier includes the same ID in the reply message. The requestor uses this ID to match replies to pending requests.

Requestor

correlation_id = str(uuid.uuid4())

reply_queue = f"reply-{correlation_id}"

message = {

'correlation_id': correlation_id,

'payload': {'user_id': '123', 'amount': 100}

}

request_queue.send(message)

Wait for reply on the reply queue

reply = reply_queue.receive(timeout=30)

Replier

def handle_request(message):

result = process(message['payload'])

reply = {

'correlation_id': message['correlation_id'],

'payload': result

}

reply_queue.send(reply)

Use Cases

Remote procedure calls. Service A asks Service B to compute something and return the result. Unlike HTTP RPC, the messaging-based approach allows the requestor to be decoupled in time -- it can send the request and check for the reply later.

Long-running operations. Submit a job, get back a job ID, check progress, and eventually receive the result. The reply might arrive minutes or hours later.

Pub/Sub vs. Request-Reply

| Aspect | Pub/Sub | Request-Reply | |--------|---------|---------------| | Communication | One-to-many | One-to-one | | Response expected? | No | Yes | | Coupling | Very loose | Moderate | | Use case | Event notification, broadcasting | Remote procedure, query | | Message ordering | Per partition/topic | Per conversation | | Error handling | Dead letter queue | Timeout + retry |

Choosing a Message Broker

Apache Kafka: Best for high-throughput event streaming, log aggregation, and event sourcing. Messages are persisted and replayable. Excellent for building event-driven architectures.

RabbitMQ: Best for traditional messaging with complex routing (direct, topic, headers, fanout exchanges). Good for request-reply patterns. Lower throughput than Kafka but richer routing features.

Amazon SQS/SNS: Fully managed. SQS for request-reply, SNS for pub/sub. No infrastructure to manage. Good for AWS-native applications.

Google Cloud Pub/Sub: Fully managed, global. Good for Google Cloud-native applications.

Advanced Patterns

Dead letter queues. When a message cannot be processed (after retries), move it to a dead letter queue for manual inspection. Prevents poison messages from blocking the main queue.

Message bridging. Forward messages between different messaging systems. For example, consume from SQS and publish to Kafka for long-term storage.

Publisher confirms. In Pub/Sub, some brokers support acknowledgments from publishers to confirm the message was received. Use this for at-least-once delivery guarantees.

Common Pitfalls

Message ordering. Distributed messaging systems rarely guarantee total order. Kafka guarantees order within a partition, not across partitions. Design your system to handle out-of-order messages.

Idempotency. Messages may be delivered more than once. Ensure your message handlers are idempotent. Use idempotency keys or deduplication.

Monitoring. Monitor queue depth, consumer lag, and processing time. Set up alerts for growing backlog.

Summary

Pub/Sub and Request-Reply serve different purposes. Use Pub/Sub when you need to notify multiple consumers of an event. Use Request-Reply when a service needs a response. Many systems use both patterns together. Choose your message broker based on throughput requirements, routing complexity, and operational preferences. Always design for idempotency and handle duplicate messages gracefully.