# Receive a recovery rehearsal as a delivery pod

Original LockedIn Labs facilitator guide · 90 planned minutes, excluding preparation and breaks · Three independent Python proof labs. This is a team practice in receiving evidence and unresolved work. It creates no production recovery decision, staffing assignment, qualification or placement outcome.

[Review the prepared recovery session](/training/guides?starter=recovery-receiving#guide-session-title). The link opens a reference to the current agenda; it does not apply it. Preview setup, timing and receiving prompts before choosing **Use this starter session**. Save an existing editable plan first if you need to retain it. The planner can download the agenda, editable planning JSON and a separate blank individual worksheet. Each person retains and fills their own worksheet; the planning JSON contains no completed participant evidence.

## 1. Prepare a bounded recovery question

A pod must explain more than whether a process ran successfully. Did an external effect already commit? Which request or ownership record survived? What would a receiving owner need before another action? Practice those questions with three separate local experiments, then return a useful evidence request.

Read the [outbox recovery lesson](/training/guides/action-recovery#run-the-two-database-proof-lab), [worker-fencing lesson](/training/guides/worker-fencing) and [application-restore lesson](/training/guides/restore-replay) beforehand. Prepare Python 3.11+ and download these original files into a disposable folder:

- [Outbox crash and destination replay](/templates/fde-outbox-replay.py)
- [Worker leases, destination fencing and operation receipts](/templates/fde-worker-fencing.py)
- [Application restore and destination replay](/templates/fde-restore-replay.py)

The first two own local child processes; all three use temporary SQLite files and synthetic effects. These selected modes make no network or model calls and require no customer data, keys or cloud account. Inspect the files and confirm the selected modes are suitable for your environment. If execution cannot be prepared, record reading/review mode and the missing setup; do not write that someone reproduced a result.

The labs reset independently. They exchange no files, tokens or operation history. A useful result in one is not an end-to-end deployment proof for another. Prepare reading and setup outside the ninety-minute practice boxes.

## 2. Rotate the receiving roles

Use three participants with synthetic aliases A, B and C; add a separate facilitator if available. Everyone keeps their own predictions, observed evidence, help and corrections. The observer records whether the receiver located the evidence without takeover. Smaller groups may combine roles, but must name the combination; solo work is self-review.

| Round | Builder | Receiver | Observer | What changes |
| --- | --- | --- | --- | --- |
| 1 · Lost response | A | B | C | Compare a duplicate-producing receiver with a guarded replay |
| 2 · Late worker | B | C | A | Keep three units fixed; change destination fencing/receipt behavior |
| 3 · Earlier restore | C | A | B | Keep three units fixed; change receipt retention |

The builder explains one prediction and locates an original result. The receiver executes the comparison themselves and explains the actual rows. The observer preserves assistance, missed steps and any correction. A polished team explanation cannot substitute for each person's account.

Application, destination and recovery/platform owners are **receiving responsibilities** in this rehearsal. Agree who will carry each question to the real owner later; practice hats grant no system access. The pod lead coordinates the open questions and bounded next actions, while the business/service owner retains any real acceptance decision.

## 3. Follow the ninety-minute running order

| Planned offsets | Activity | Evidence the receiver must locate |
| --- | --- | --- |
| 0–15 | Round 1 practice | Pending/unknown caller after a real child exit; destination effect and receipt |
| 15–25 | Round 1 receiving review | Naive versus guarded redelivery; local acknowledgement versus effect count |
| 25–40 | Round 2 practice | Old worker's refused queue finish versus actual destination effects |
| 40–50 | Round 2 receiving review | Six units versus three; a receipt does not retroactively authorize a stale worker |
| 50–70 | Round 3 practice | Completed reference, restored pending row and retained destination effect |
| 70–80 | Round 3 receiving review | Two effects with one replacement receipt versus one effect with retained history |
| 80–90 | Receiving-owner challenge | Unresolved history, owner and evidence request; a real acceptance decision remains separate |

The planner represents the last two rows as one twenty-minute receiving-review box. These are planned offsets, not measured response times or promised learning outcomes. Add preparation, breaks and any extra coaching separately. If a step is unrun, keep it unrun and agree a next task rather than compressing the evidence into a completed label.

## 4. Run the three receiving comparisons

Keep the original prediction before opening each report. Save reports under different names and record the exact file/runtime version used.

### Round 1: the caller is uncertain after a committed effect

```sh
python3 fde-outbox-replay.py --case naive > round1-naive.json
python3 fde-outbox-replay.py --case guarded > round1-guarded.json
```

Receiver B locates `caller_after_crash`, `destination_observer_after_crash`, `caller_after_replay` and `destination_observer_after_replay`. The caller begins pending with unknown outcome in both runs, while the destination observer can inspect one committed effect. Naive replay leaves two effects; guarded replay leaves one and returns a retained receipt. A destination observer's evidence is not automatically knowledge the caller already possessed. Observer C records B's actual lookup and hints.

### Round 2: queue completion and destination authority are separate

```sh
python3 fde-worker-fencing.py --case fence-old-first --units 3 > round2-fence.json
python3 fde-worker-fencing.py --case fence-and-receipt --units 3 > round2-receipt.json
```

Receiver C locates the old worker's `queue_finish`, actual destination attempts and `final_destination`. Old-first fencing can admit six units even though the old queue finish is refused. The combined receipt candidate retains three units and one receipt. That receipt prevents another local effect for the same operation; it does not retroactively authorize the stale worker or prove a real identity/lease service. Observer A keeps the first explanation and any correction separate.

### Round 3: an earlier application copy can resurrect completed work

```sh
python3 fde-restore-replay.py --case pruned-receipt --units 3 > round3-pruned.json
python3 fde-restore-replay.py --case retained-receipt --units 3 > round3-retained.json
```

Receiver A locates `original_application_after_completion`, `restored_application_before_retry`, `destination_before_retry` and `final_destination`. The restored application is pending in both runs. Pruning the old receipt leaves the original effect; the unsafe retry creates a replacement receipt and a second effect, totaling six units. Retained history replays three units with one effect. One final receipt is not evidence of one actual effect. Receipt pruning is deliberate SQL deletion, not a tested expiry timer. Observer B records A's commands, row references and assistance.

## 5. Return a question when history is unavailable

At minute 80, the facilitator introduces this counterfactual: “Your local comparison is understood. The real destination's authoritative history is unavailable, and nobody has confirmed its receipt retention. Can we resume the restored jobs?” No actual provider failure or external inquiry occurs here.

The receiver should preserve the uncertainty. A missing receipt is not proof an operation never happened, and inventing a new ID can repeat the same intended work. Record an authoritative status/reconciliation request with the destination owner. Ask the application owner which stable operations and payloads can reappear, and the recovery/platform owner which data histories were restored or retained. The pod lead names a bounded follow-up, escalation role and proposed update time to agree. Do not invent an accepted SLA, permission or available owner.

Then challenge the team's language: “Can we report exactly-once execution and an achieved production Recovery Time Objective (RTO)?” Keep the observed local backup, process exit, counter and receipt facts. These independent labs establish neither a distributed production guarantee nor service recovery time. The real service/business owner must define useful-service acceptance with the actual authorized operators.

## 6. Keep individual evidence and the receiving decision separate

Open **Prepare individual practice worksheets** in the planner, choose **Download blank worksheet (.md)** and give each person a separate local copy. Use its blank prediction, own contribution, observed result, assistance, changed condition and receiving-feedback fields. The shared agenda is an instruction, not anyone's completed record. A facilitator may retain a synthetic observation summary only with the participants' agreed handling; keep client/employee records and credentials out of the exercise.

Use this additional closing record alongside the individual worksheets:

```text
Synthetic session / exact lab files and runtime versions:
A's own receiving action / evidence pointer / help / correction:
B's own receiving action / evidence pointer / help / correction:
C's own receiving action / evidence pointer / help / correction:
Unrun or combined-role observations:
Application owner question / role to confirm / next evidence:
Destination owner question / role to confirm / next evidence:
Recovery/platform question / role to confirm / next evidence:
Pod follow-up coordinator / bounded action / update time to agree:
Real service acceptance: not decided by this rehearsal
```

Repeat only an unclear receiving step with a changed condition and retain the first observation. Do not average the three reports into a qualification score. The next useful result is a more precise, reproducible explanation and an accountable evidence request; independent client work and pod-lead authority require separate real-world evidence and human decisions.