Dead-letter queues: where failed messages should go
Short answer: A dead-letter queue (DLQ) is a separate queue where messages go after they've failed processing a set number of times. Instead of retrying a broken message forever, blocking other work or silently dropping it, you park it somewhere visible, alert on it, fix the cause and then move it back for reprocessing. Set a sensible maximum number of attempts, make the DLQ keep messages longer than the source queue, alert whenever it isn't empty, and have a written way to inspect and redrive messages.
The problem: poison messages
Queues are how many business apps hand work to the background: sending order confirmations, syncing stock to marketplaces, generating invoices, resizing images, posting to an accounting system. Most messages succeed. Some fail for a moment, because an API was down or a database briefly busy, and succeed on retry.
A few fail every time. A malformed payload, a product that's been deleted, an address the shipping API rejects. These are often called poison messages. Without a plan, a poison message either:
- Retries forever, using resources and, on some queue types, holding up messages behind it, or
- Gets dropped, so a customer's invoice or shipment quietly never happens.
A dead-letter queue is the middle path: stop retrying, but keep the message where someone will see it.
How DLQs work in common systems
The idea is the same across queue services, though the details differ.
In Amazon SQS, the documentation explains that you attach a redrive policy to a source queue with a maxReceiveCount: the number of times a consumer can receive a message before it moves to the dead-letter queue. SQS also offers a redrive feature to move messages back out of the DLQ once you've fixed the problem.
In RabbitMQ, you set a dead-letter exchange on a queue with the x-dead-letter-exchange argument. Its documentation lists when messages are dead-lettered: when a consumer rejects them without requeueing, when they expire because of a message TTL, when the queue exceeds a length limit, or when a quorum queue's delivery limit is exceeded.
Many background-job libraries have their own version, often called a "dead" or "failed jobs" set. Check what yours provides before building one.
Choosing the retry limit
The maximum number of attempts should be high enough to ride out temporary problems and low enough that genuine poison messages leave quickly. A few practical points:
- Combine retries with increasing delays, so a short outage at a supplier's API doesn't exhaust all attempts in seconds.
- Distinguish errors in your code where you can. A validation error that will never succeed can go straight to the DLQ; a timeout deserves retries.
- Consider how long a message is invisible between attempts, so the total retry window matches how long outages typically last for your integrations.
There's no universal right number. Start with a handful of attempts and adjust based on what ends up in the DLQ.
Watch the retention period
Messages don't live forever in queues. The SQS documentation highlights a detail that catches people out: for standard queues, a message's expiry is based on its original enqueue time, which doesn't change when it moves to the DLQ. Amazon therefore recommends setting the DLQ's retention period longer than the source queue's. Otherwise a message might expire from the DLQ shortly after arriving, before anyone looks.
The same documentation notes that for FIFO queues the timestamp resets on the move, and cautions against using a DLQ with FIFO queues if breaking the exact order of messages would cause problems.
Whatever system you use, check how long dead messages are kept and make sure it's longer than your team's realistic response time, including weekends.
Alert on anything in the DLQ
A dead-letter queue nobody watches is just a slower way of losing messages. Set an alert for when the DLQ's message count goes above zero, or above a small threshold for very busy systems. Route it to the people who own the integration, with enough context to act:
- Which queue and how many messages.
- The error from the last attempt, if your consumer records it.
- A link to wherever messages can be inspected.
Make sure the consumer logs the failure reason with an identifier, such as the order ID, so you can find the cause without decoding raw messages.
Inspect, fix, redrive
When messages land in the DLQ, follow a simple routine:
- Look at a sample. What do the failing messages have in common?
- Fix the cause. Correct the code, the data or the configuration.
- Redrive. Move the messages back to the source queue to be processed again.
- Confirm. Check they succeeded and the DLQ is empty.
Before redriving, make sure your consumer is idempotent, meaning processing the same message twice is harmless. Some messages may have partly succeeded on earlier attempts.
Occasionally a message should be discarded, for example an update for a product that no longer exists. Record why, so the decision is traceable.
A worked example
Take a hypothetical Adelaide wine retailer whose app queued orders for sync to its warehouse system. A change to the warehouse API rejected orders with gift messages over a certain length, and those messages retried until they expired. Nobody noticed until customers asked where their parcels were. The team added a DLQ after five attempts with increasing delays, set its retention longer than the main queue's, alerted on any message in it and logged the order ID and API error. The next rejection was spotted within the hour, the message length was trimmed in code, and the orders were redriven.
FAQ
Is a DLQ the same as retrying?
No. Retries happen before a message reaches the DLQ. The DLQ is where it goes when retries run out.
Should every queue have a DLQ?
Most queues carrying business work should. For purely disposable messages, dropping after retries may be acceptable.
Can I redrive automatically?
You can, but only after fixing the cause. Automatic redrive of an unfixed problem just repeats the failure.
If background messages in your app sometimes vanish or retry endlessly, ask us to set up proper failure handling.