# Recover quarantined work without replaying the incident

Original LockedIn Labs instruction · 50-minute recovery and receiving lab · reviewed 27 October 2026.

A queue is now quiet. Is the incident resolved, or did its unfinished work simply
move somewhere else? You will inspect a failed delivery, keep its original
payload, change a reader, and replay one item against actual committed SQLite
rows. A second incident has already written its effect before settlement fails.
Its replay needs a different diagnosis.

This standalone workshop builds on basic transactions and operation identity.
Start with [Queue 101](/training/guides/queues) if receive, acknowledgement and
redelivery are new. You need Python 3.11+ and the standard library. There is no
cloud account, broker, credential, model or production operation.

## 1. Separate the three receiving questions

A **poison message** repeatedly fails a particular consumer: its schema may be
unsupported, its input invalid, or the consumer defective. A **dead-letter queue
(DLQ)** retains deliveries that were set aside for investigation. **Redrive** or
**replay** makes selected retained work available for another processing attempt.
Those words do not mean the original fault was repaired.

| Question | Evidence to inspect | What it cannot establish alone |
| --- | --- | --- |
| Did delivery stop retrying? | Delivery state, attempt history, reason | Whether a business effect exists |
| Did work leave quarantine? | The move and its new delivery identity | Whether the new consumer succeeded |
| Did the intended action happen? | Matching operation receipt, payload and effect | That every other unresolved item is repaired |

An application operation key describes intended business work. A broker message
ID identifies transport work under that broker's contract. A new delivery ID can
still refer to the original operation. Replacing its business key just to make a
replay look new can create another effect.

A known unsupported schema is a deterministic incompatibility. A production handler should classify it and route it for correction instead of repeatedly retrying unchanged input. This lab deliberately spends a finite allowance on that error to expose what happens when replay resets the counter.

A budget also needs a scope. Three attempts per delivery do not mean three
attempts across an incident if every redrive creates a fresh allowance. Preserve
the earlier attempts, set an incident-wide decision boundary and name the person
who may authorize another bounded replay.

## 2. Draw movement separately from completion

In this lab, JOB-1 and JOB-2 request fictional maintenance drafts. Reader 1
understands schema 1. The first incident supplies schema 2 for JOB-1; reader 2
adds support for that same payload. JOB-2 remains ordinary schema-1 work.

The second incident uses schema 1 for both jobs. JOB-1's effect commits, then the
fixture omits settlement on each original attempt. Subsequent attempts find and
reuse its receipt, but that delivery still exhausts its settlement attempts.
Quarantine therefore contains work whose effect already exists.

The visual has six authored frames grounded in the downloadable observations.
It does not execute a queue in your browser. Read the written ledger below as
well as the map, and predict where each number comes from.

## 3. Run three independently reset incidents

Download [the queue recovery runner](/templates/fde-queue-recovery.py). Keep
reports in a new local folder. The program accepts no database paths or hosts;
its separate temporary SQLite file for each incident is removed when it exits.

```sh
python3 --version
python3 fde-queue-recovery.py --all > recovery-original.json
python3 fde-queue-recovery.py --case settlement-fixed --replay-key new > recovery-new-key.json
python3 fde-queue-recovery.py --case unchanged --attempt-limit 2 > recovery-smaller-budget.json
```

The queue is an explicit single-worker delivery model. It uses no broker,
visibility timer, sleep, actual process crash or authentication. Claims, attempt
records, quarantine history, business effects and receipts are real SQLite
transactions. The effect and receipt share one transaction; settlement is a
later transaction, even though this teaching model uses one file. Actual remote
settlement and business writes usually have separate owners and failure modes.

`initial`, `before_replay`, `after_move_before_work` and `final` are fresh reads
of committed tables. `attempts` retains every attempt, reader version, actual
apply/reuse result, effect ID, error and settlement result. The JSON preserves
original bodies and delivery IDs rather than showing only totals.

### The written ledger

With the default three-attempt limit:

| Incident | Before replay | Immediately after move | Final evidence |
| --- | --- | --- | --- |
| `unchanged` | JOB-1 quarantined for unsupported_schema; JOB-2 has one effect | Quarantine count is zero; replay has not run | Reader 1 fails three more times; one effect and a second quarantine record remain |
| `reader-fixed` | Same unsupported schema and one JOB-2 effect | Same movement, same original body | Reader 2 processes JOB-1 once; two effects, two matching receipts, retained original quarantine record |
| `settlement-fixed` | JOB-1 applied once then reused twice; JOB-2 applied once; JOB-1 delivery quarantined | Quarantine is zero, but both effects already existed | Original-key replay reuses JOB-1's receipt and settles; total effects stays two |

Inspect delivery 1, its replay delivery 3 and their identical body strings.
The original state becomes `moved`, not retroactively `settled`. Its quarantine
record stays in `quarantine_history`. JOB-2 receives only once after the failing
item reaches its bound; this model does not prove fairness or throughput.

Change only `--replay-key new` in the settlement incident. Delivery 3 now uses
JOB-1-NEW with the same body. It creates effect 3 instead of reusing effect 1.
Both JOB-1 effects describe the same synthetic draft request. All active delivery
queues can be empty while duplicate business work remains. The new key does not
repair or reverse either original effect.

Reduce `--attempt-limit` to 2. The unchanged incident now records two original
failures and two replay failures, plus one successful JOB-2 attempt. The new
limit reduces attempted work; it still does not make schema 2 readable. No
automatic replay loop follows the second quarantine.

## 4. Plan a canary replay the receiver can stop

A canary is a small, explicitly selected first replay whose evidence you inspect
before expanding the operation. In this model it is exactly one retained item.
In an enterprise, agree the selection, rate, maximum work, stop conditions,
permissions and owner before moving anything. A producer fix, reader fix and
payload correction are different changes; retain the original message and the
approved transformation if one is necessary.

| Receiving role | Concrete responsibility |
| --- | --- |
| Application owner | Explain the failure class and exact reader change; verify payload compatibility and operation receipt retention |
| Platform operator | Verify current queue access, retention, replay selection/rate and stop route; preserve original diagnostics |
| Workflow owner | Decide whether the original action is still wanted and who resolves an existing or extra effect |
| Receiving engineer | Reproduce one selected item, inspect its matching result and unresolved work, then return a bounded decision |

No stored message carries perpetual permission. Recheck current authority,
expiry and business relevance. This fixture has no identity service or retention
expiry, so those checks require evidence from the actual enterprise contract.
For an external email, payment or clinical action, a local SQLite receipt cannot
make the remote effect atomic. Use the destination's reconciliation contract.

## 5. Compare provider contracts before adopting the model

Amazon SQS documents a receive-count policy for dead-lettering. Its managed
redrive assigns new message IDs and enqueue times; it does not filter or modify
messages during the move. That makes a separate stable application identity
especially important. Start a redrive conservatively and observe downstream
capacity rather than treating movement speed as a recovery metric.

Azure Service Bus offers explicit dead-letter reasons and a maximum delivery
count under Peek-Lock. Its DLQ does not automatically clean itself up, and TTL
is not observed there. Do not import that retention behavior into SQS or assume
that completing a dead-letter message proves the intended action succeeded.
The lab's attempt counter, one-file queue and explicit one-item move are its own
teaching contract, not an SDK implementation of either service.

Primary references checked 27 October 2026:

- [SQS dead-letter behavior and retention](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html).
- [SQS redrive identity, rate and movement limits](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue-redrive.html).
- [Service Bus dead-letter reasons, settlement and retention](https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dead-letter-queues).

## 6. Run the receiving review as a pod

Use three people where possible: application owner, operator and receiver.
Combine roles explicitly for smaller groups; a solo attempt is self-review.
The application owner explains one report. The receiver chooses a different
case or attempt limit, predicts it first and reproduces it without author
takeover. The operator points to the retained attempt and quarantine records,
then states the exact stop condition for any additional replay.

Keep one record per participant. The team outcome does not replace individual
evidence:

```text
My role / report and exact command:
Prediction before the original run:
Observed delivery IDs / attempts / reader version:
Observed business keys / receipt and effect IDs:
One changed condition / prediction before that run:
New observations kept separately:
Receiving action actually attempted / assistance:
What remains unresolved / accountable role / next evidence:
```

Interview practice: why can an empty DLQ coexist with both missing work and
extra work? Where does the replay attempt budget reset? Which evidence would
you require before widening a canary? Explain your answer using particular
rows, not “exactly once” or “the queue handles it.”

Continue to [HTTP outcome reconciliation](/training/guides/http-outcomes) for an
actual transport fault, or use the protected [recovery lesson](/app/academy?course=enterprise-integration&lesson=timeouts-and-retries&view=focus)
with existing training access. Completing practice is not an independent review,
a production replay authorization or a placement decision.