Recover quarantined work without replaying the incident

A poison message repeatedly fails a particular consumer: its schema may be unsupported, its input invalid, or the consumer defective. A dead-letter queue (DLQ) retains deliveries that were set aside for investigation. Redrive or replay makes selected retained work available for another processing attempt. Those words do not mean the original fault was repaired.

An application operation key describes intended business work. A broker message ID identifies transport work under that broker's contract. A new delivery ID can still refer to the original operation. Replacing its business key just to make a replay look new can create another effect.

A known unsupported schema is a deterministic incompatibility. A production handler should classify it and route it for correction instead of repeatedly retrying unchanged input. This lab deliberately spends a finite allowance on that error to expose what happens when replay resets the counter.

A budget also needs a scope. Three attempts per delivery do not mean three attempts across an incident if every redrive creates a fresh allowance. Preserve the earlier attempts, set an incident-wide decision boundary and name the person who may authorize another bounded replay.

In this lab, JOB-1 and JOB-2 request fictional maintenance drafts. Reader 1 understands schema 1. The first incident supplies schema 2 for JOB-1; reader 2 adds support for that same payload. JOB-2 remains ordinary schema-1 work.

The second incident uses schema 1 for both jobs. JOB-1's effect commits, then the fixture omits settlement on each original attempt. Subsequent attempts find and reuse its receipt, but that delivery still exhausts its settlement attempts. Quarantine therefore contains work whose effect already exists.

The visual has six authored frames grounded in the downloadable observations. It does not execute a queue in your browser. Read the written ledger below as well as the map, and predict where each number comes from.

Download the queue recovery runner. Keep reports in a new local folder. The program accepts no database paths or hosts; its separate temporary SQLite file for each incident is removed when it exits.

python3 --version
python3 fde-queue-recovery.py --all > recovery-original.json
python3 fde-queue-recovery.py --case settlement-fixed --replay-key new > recovery-new-key.json
python3 fde-queue-recovery.py --case unchanged --attempt-limit 2 > recovery-smaller-budget.json

The queue is an explicit single-worker delivery model. It uses no broker, visibility timer, sleep, actual process crash or authentication. Claims, attempt records, quarantine history, business effects and receipts are real SQLite transactions. The effect and receipt share one transaction; settlement is a later transaction, even though this teaching model uses one file. Actual remote settlement and business writes usually have separate owners and failure modes.

initial, before_replay, after_move_before_work and final are fresh reads of committed tables. attempts retains every attempt, reader version, actual apply/reuse result, effect ID, error and settlement result. The JSON preserves original bodies and delivery IDs rather than showing only totals.

The written ledger

With the default three-attempt limit:

Inspect delivery 1, its replay delivery 3 and their identical body strings. The original state becomes moved, not retroactively settled. Its quarantine record stays in quarantine_history. JOB-2 receives only once after the failing item reaches its bound; this model does not prove fairness or throughput.

Change only --replay-key new in the settlement incident. Delivery 3 now uses JOB-1-NEW with the same body. It creates effect 3 instead of reusing effect 1. Both JOB-1 effects describe the same synthetic draft request. All active delivery queues can be empty while duplicate business work remains. The new key does not repair or reverse either original effect.

Reduce --attempt-limit to 2. The unchanged incident now records two original failures and two replay failures, plus one successful JOB-2 attempt. The new limit reduces attempted work; it still does not make schema 2 readable. No automatic replay loop follows the second quarantine.

A canary is a small, explicitly selected first replay whose evidence you inspect before expanding the operation. In this model it is exactly one retained item. In an enterprise, agree the selection, rate, maximum work, stop conditions, permissions and owner before moving anything. A producer fix, reader fix and payload correction are different changes; retain the original message and the approved transformation if one is necessary.

No stored message carries perpetual permission. Recheck current authority, expiry and business relevance. This fixture has no identity service or retention expiry, so those checks require evidence from the actual enterprise contract. For an external email, payment or clinical action, a local SQLite receipt cannot make the remote effect atomic. Use the destination's reconciliation contract.

Amazon SQS documents a receive-count policy for dead-lettering. Its managed redrive assigns new message IDs and enqueue times; it does not filter or modify messages during the move. That makes a separate stable application identity especially important. Start a redrive conservatively and observe downstream capacity rather than treating movement speed as a recovery metric.

Azure Service Bus offers explicit dead-letter reasons and a maximum delivery count under Peek-Lock. Its DLQ does not automatically clean itself up, and TTL is not observed there. Do not import that retention behavior into SQS or assume that completing a dead-letter message proves the intended action succeeded. The lab's attempt counter, one-file queue and explicit one-item move are its own teaching contract, not an SDK implementation of either service.

Primary references checked 27 October 2026:

• SQS dead-letter behavior and retention.
• SQS redrive identity, rate and movement limits.
• Service Bus dead-letter reasons, settlement and retention.

Use three people where possible: application owner, operator and receiver. Combine roles explicitly for smaller groups; a solo attempt is self-review. The application owner explains one report. The receiver chooses a different case or attempt limit, predicts it first and reproduces it without author takeover. The operator points to the retained attempt and quarantine records, then states the exact stop condition for any additional replay.

Keep one record per participant. The team outcome does not replace individual evidence:

My role / report and exact command:
Prediction before the original run:
Observed delivery IDs / attempts / reader version:
Observed business keys / receipt and effect IDs:
One changed condition / prediction before that run:
New observations kept separately:
Receiving action actually attempted / assistance:
What remains unresolved / accountable role / next evidence:

Interview practice: why can an empty DLQ coexist with both missing work and extra work? Where does the replay attempt budget reset? Which evidence would you require before widening a canary? Explain your answer using particular rows, not “exactly once” or “the queue handles it.”

Continue to HTTP outcome reconciliation for an actual transport fault, or use the protected recovery lesson with existing training access. Completing practice is not an independent review, a production replay authorization or a placement decision.