Message Queues: Producer-Consumer Model & Competing Consumers

The Producer-Consumer pattern decouples work generation from work execution through an intermediate buffer:

• Producer: Any application service that constructs a structured message (JSON, Protocol Buffers, or Avro) and enqueues it via an API call (SendMessage in SQS or basic.publish in RabbitMQ).
• Message Queue: A distributed, durable store that holds messages in sequential order, managing delivery state, timeouts, and replication across multiple Availability Zones.
• Consumer (Worker): A background process that polls or receives messages, executes business logic, and notifies the queue upon completion.

In point-to-point queues, each message is delivered to and processed by exactly one consumer, distinguishing it from broadcast publish-subscribe models.

To process heavy workloads, multiple identical worker instances subscribe to the same queue. This is known as the Competing Consumers Pattern:

• Dynamic Load Distribution: Workers pull messages from the queue independently. Faster workers naturally pull more messages, while slower workers handle fewer, creating automatic load balancing without a central coordinator.
• Queue-Depth-Driven Auto-Scaling: Instead of scaling worker fleets based on CPU utilization (which can be misleading for I/O-bound tasks), systems scale on Backlog Depth per Worker:

Target Workers = ≤ft\lceil \frac{Queue Depth (ApproximateNumberOfMessages)}{Target Processing Rate per Worker × Acceptable Latency SLA} \right\rceil

For example, if the queue holds 12,000 messages, each worker handles 10 msgs/sec, and messages must clear within 60 seconds, the auto-scaler provisions \lceil 12,000 / (10 × 60) \rceil = 20 workers.

When a worker retrieves a message, the queue cannot immediately delete it—if the worker crashes during execution, the message would be permanently lost. Instead, distributed queues use Visibility Timeouts:

1. State Transition to In-Flight: When Worker A dequeues Message 1, the queue hides Message 1 from all other workers for a configured Visibility Timeout window (e.g., 30 seconds).
2. Positive Acknowledgment (ACK): Worker A finishes processing in 4 seconds and issues a DeleteMessage call with the unique ReceiptHandle. The queue permanently purges the message.
3. Negative Acknowledgment (NACK / Reject): If Worker A detects a transient failure (e.g., downstream database temporarily locked), it sends a NACK, causing the queue to immediately reset visibility so another worker can retry without waiting for the timeout.
4. Worker Crash & Automatic Failover: If Worker A dies from an Out-Of-Memory (OOM) error or hardware failure, no ACK is received. Once the 30s timeout elapses, Message 1 automatically transitions back to visible state. Worker B polls the queue, receives Message 1 with an incremented ReceiveCount = 2, and completes the task.

Visibility Timeout Sizing & Heartbeating

Setting the visibility timeout too short causes duplicate processing (Worker B starts processing Message 1 while Worker A is still working on it). Setting it too long delays crash recovery.

• Rule of Thumb: Configure default visibility timeout to 3× your P_{99} task execution time.
• Heartbeat Lease Renewal: For tasks with variable execution times (e.g., video encoding taking anywhere from 10 seconds to 10 minutes), the worker runs a background heartbeat thread. Every 15 seconds, it calls ChangeMessageVisibility(visibilityTimeout = 30s) to extend its lease until completion.

Short Polling vs Long Polling

• Short Polling: The queue samples a subset of storage servers and returns immediately, even if no messages are found. This generates millions of empty HTTP 200 OK calls, driving up cloud bills.
• Long Polling (WaitTimeSeconds = 20): The queue connection remains open for up to 20 seconds until a message arrives, reducing empty responses by over 90\% and reducing delivery latency from seconds to milliseconds.