When One Message Keeps Killing the Worker: A Rehearsal for Quarantine Without Blind Retries

A worker that crashes mid-message and restarts will usually receive the same message again. If the message is the cause, the loop never ends and everything behind it waits. This article proposes a rehearsal you can run in a test environment to check that your design contains such a message.

Core rules of the design

1. Record the attempt before processing. Increment a durable attempt counter for the message first, then process. A crash cannot erase an attempt that was written earlier. If the counter is only updated on failure handling, a hard crash skips it.
2. Set a small attempt limit, for instance 3, as a stated assumption you can change.
3. At the limit, move the message to a quarantine area with its content, attempt count and last known stage. Then continue with the rest of the queue.
4. Quarantine is not deletion and not automatic replay. A person decides what happens next.

Decision branches

Branch A, attributable crash: only one message was in flight, so the crash belongs to it. Count it.

Branch B, unattributable crash: several messages were in flight. Do not blame them all. Drop concurrency to one for a trial period so the next crash points at a single message. Stop this trial if no crash recurs within a stated number of messages, and restore concurrency.

Branch C, shared dependency failure: the crash or error is a downstream outage rather than the message. If several different messages fail close together, pause consumption instead of counting attempts against each. Quarantining healthy messages during an outage is the wrong response.

Branch D, uncertain external write: the worker reached the point of calling an outside system and died before recording the result. The outcome is unknown. Do not repeat that write automatically, whatever the attempt count. Mark the message 'outcome unknown' and route it to a person who checks the outside system first. A local claim that you started the work is not proof it was done or not done.

Worked hypothetical rehearsal

Queue of five messages M1 to M5, concurrency one, limit 3. M3 contains an input that crashes the parser before any external call.

Totals to expect: 7 attempts started, 4 completed, 1 quarantined, 0 left in queue. M4 and M5 must complete without waiting for a human.

Variant: if M3 had crashed after sending an external request, expect 1 quarantined with the label 'outcome unknown' after the first crash following the restart, not three repeats of the write. Totals then: 5 attempts started if M3 stops at 1, with the same 4 completed.

Stop conditions

• Stop the rehearsal and the live consumer if the quarantine area itself fills or cannot be written; continuing would lose messages.
• Stop automatic quarantine if more than a stated share of recent messages are being quarantined, since that suggests Branch C rather than poison.
• Stop any replay from quarantine until the cause is identified.

Proposed acceptance tests (to be performed by you, none performed here)

1. Kill the worker at a point after the counter write: the counter still shows the attempt.
2. The rehearsal totals above hold exactly.
3. During a simulated dependency outage across several messages, no message is quarantined and consumption pauses.
4. In the unknown-outcome variant, no second external request is observed.
5. Quarantined content is readable by an operator but not replayed unless an explicit action is taken.

These are conditional design suggestions. Whether your queue exposes delivery counts or crash signals depends on your own setup.

Sources