Messages that keep failing go to a dead-letter queue
A message the consumer cannot process is set aside after a few attempts, with an alert, so it neither blocks the queue nor disappears.
HighChecked by hand
- Applies to:
- Services that consume events
- Rule id:
- events.dead-letter-queue
Why it matters
Some messages will never succeed: a payload the code can't parse, a reference to something that doesn't exist. Retried forever, one such message blocks everything behind it. Dropped silently, it is lost business data. A dead-letter queue keeps the flow moving and keeps the message for someone to look at.
How to apply it
- Retry a failing message a small, fixed number of times with backoff, then move it to a dead-letter queue.
- Keep the original payload, the error and the attempt count with it.
- Alert when the dead-letter queue is not empty. A queue nobody watches is a slow way to lose data.
- Have a way to replay messages from it once the cause is fixed. Replay relies on the consumer handling duplicates.
- Tell apart errors worth retrying (a timeout) from ones that never will be (a validation error), and skip the retries for the second kind.
Example
The consumer retries OrderPlaced three times over a minute, then publishes it to orders.shipping.dlq with the stack trace in a header. An alert fires when the queue holds any message for five minutes; a runbook entry explains how to replay.
References
Checklists with this rule
Related agent skill: async-messaging