# Restore the data without repeating completed work

Original LockedIn Labs proof workshop · 50 planned minutes, excluding preparation · Python 3.11+ standard library. Back up and restore an actual scratch application database while a separate scratch destination retains its committed effects. Five controlled comparisons use synthetic counters. No account, network, real tool action, production restore or service outage is involved.

## 1. Understand the recovery boundary

A backup preserves an earlier database state. Restoring that state can bring back pending work that the live application later completed. An effect committed in another system does not disappear because your application returned to an older point.

Imagine a job asking another service to allocate four synthetic units. The backup says pending. Later, the destination applies four units and the application records completion. Restore the earlier application copy: its schema is valid and its read query works, but it once again says pending. A worker that treats this row as proof of unfinished business can perform the effect again.

This is separate from losing recent data. Recovery Point Objective (RPO) concerns the recoverable data window; Recovery Time Objective (RTO) concerns interruption before agreed useful service returns. Those targets also need a business acceptance boundary. Neither a readable restored file nor a fast restore establishes which external operations already occurred. This workshop measures neither target.

## 2. Follow one operation across two histories

The application owns `JOB-1`, carrying business operation `OP-1` and its units. A destination owns the effect rows. In the guarded comparisons, it also owns an operation receipt bound to those units. An identical retry returns the stored result; a different payload for the same operation conflicts.

The destination commits effect and receipt in one SQLite transaction. No transaction spans application and destination files. The runner creates an application backup before dispatch, executes the original destination action, records application completion, then restores the earlier backup into a new application file. The completed original remains available for comparison; the destination is retained.

The mechanism below is an authored explanation. The runner's actual rows are the evidence. Its deliberate receipt deletion represents a retained-history gap; it does not implement a timed expiry service or prescribe a universal retention period.

## 3. Predict and run five recovery cases

Download [the standalone runner](/templates/fde-restore-replay.py) into a disposable folder, inspect it and run:

```sh
python3 --version
python3 fde-restore-replay.py > restore-original.json
python3 fde-restore-replay.py --case retained-receipt
python3 fde-restore-replay.py --case pruned-receipt --units 3 > restore-pruned-three.json
```

Use `py -3` on Windows if that is your launcher. No package installation is needed. The runner accepts only a named case and original units from 1 through 9; it opens no user-selected database or URL. Each case creates and cleans up its own temporary files. Save earlier reports under distinct names.

Predict the retry and destination outcome before reading the report:

| Case | What the restored worker and destination do | Retry result | Final units / effects / receipts |
| --- | --- | --- | --- |
| `unguarded` | Restore pending; destination has no receipt contract | Applied again | 8 / 2 / 0 |
| `retained-receipt` | Preserve OP-1 and its original units; destination retains its receipt | Stored result replayed | 4 / 1 / 1 |
| `new-operation-id` | Worker invents OP-1-recovered for the same intended work | New operation applied | 8 / 2 / 2 |
| `pruned-receipt` | Delete only OP-1's destination receipt; its effect remains | Applied again | 8 / 2 / 1 |
| `changed-payload` | Reuse OP-1 but request one additional unit | Payload conflict | 4 / 1 / 1; restored job stays pending |

All five restore the same pending application state. All five pass the actual application read and local integrity checks. Those identical observations conceal different business outcomes. With `--units 3`, the duplicate cases end at six and the retained/conflict cases end at three. Changed-payload requests four in that run; it does not silently change the retained original receipt.

## 4. Inspect the evidence before resuming work

Compare `backup_job`, `original_application_after_completion` and `restored_application_before_retry`. These are actual reads: pending, completed, pending. `original_application_retained` shows that the runner did not overwrite its completed reference.

Compare `destination_before_app_restore` with `destination_after_app_restore`: both retain the original effect. The application restore did not reset this separate database. In `pruned-receipt`, inspect `receipt_pruning.deleted_rows` and the before/after receipt lists. The `effects` list is still present after deletion; no cancellation or compensation happened. The unsafe retry creates a replacement receipt storing eight units in the default run. Its one final receipt does not erase the two actual effect rows.

Then compare `retry_request`, `retry_result`, `final_restored_application` and `final_destination`. A completed restored job can coexist with two effects. A replayed result is historical evidence for the same operation, not proof of current authorization or a fresh action. A conflict leaves the original destination intact and the restored job unresolved. Do not change its ID merely to obtain a successful response.

The destination is a local counter implementation. Real APIs may define different operation scope, retention, conflict and status-query contracts. A missing receipt after history pruning cannot by itself distinguish never-executed work from old completed work. Preserve that uncertainty and require an authoritative reconciliation path before deciding to retry, compensate or accept service. This lab deliberately executes the unsafe retry so you can inspect the duplicate; it is not a recovery runbook for a real service.

## 5. Receive the recovery in a team

Allow ten minutes for a receiving review. Each person keeps a separate prediction, exact command, output pointer, own contribution, help and correction. A second engineer changes original units to three, runs `pruned-receipt` and locates both effect rows totaling six. Swap roles for `retained-receipt --units 3`: locate one effect, one receipt and the historical replay result. Solo work is self-review.

The application owner identifies stable business operations and which restored jobs might reappear. The destination owner supplies the actual receipt/status contract, payload rules and retention evidence. The recovery/platform owner identifies backup coverage, recovery points and which dependencies were retained or restored separately. The pod lead keeps unresolved operations and receiving responsibilities visible. None inherits another owner's permissions or accepts business recovery just because the runner exits successfully.

Use this blank receiving record with synthetic aliases and permitted artifacts:

```text
Person / role / exact runner version:
Prediction before execution / command:
Backup job / completed reference / restored job evidence:
Destination retained / receipt retained or removed / actual effect rows:
Changed input / actual retry result / assistance received:
Unresolved business operation / evidence still needed:
Application owner / destination owner / recovery owner:
Proposed next action / receiving question / human decision still needed:
```

For cohort practice, [follow the recovery pod facilitator guide](/training/guides/recovery-receiving). It connects three independent receiving comparisons through role swaps, a preview-only prepared agenda and blank individual worksheets.

## 6. Defend the boundary in an interview

- **Why can an integrity-checked restore repeat work?** The restored application can be older than a separate committed destination effect. Storage validity says nothing about that cross-system history.
- **Why preserve the operation ID across restore?** It lets a destination with a surviving compatible receipt recognize the same business intent. A fresh attempt ID is not a replacement for that intent.
- **Does an idempotent endpoint make every restore safe?** No. The caller must preserve scope and payload, and the relevant history must remain available under the actual contract. Identity, access and reconciliation still need evidence.
- **What if the receipt was pruned?** Missing history is insufficient evidence that the action never happened. Locate an authoritative status or reconciliation record; keep the receiving decision unresolved until supported.
- **Can we claim production RTO from this result?** No. These are sequential local file operations with a synthetic destination and no service outage or distributed recovery experiment.

After this lesson, use the [systems recovery workshop](/training/guides/systems#4-understand-both-recovery-clocks) for the two clocks and compatibility, then the [outbox recovery lab](/training/guides/action-recovery#run-the-two-database-proof-lab) for a crash between destination commit and local completion. Keep restore, crash and receipt-retention evidence distinct.

## Primary references

The [SQLite backup API](https://www.sqlite.org/backup.html) and [Python backup method](https://docs.python.org/3.11/library/sqlite3.html#sqlite3.Connection.backup) define the actual copy mechanism used here. The [AWS Builders' Library on idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) discusses caller request identity, atomic recording, changed intent and service-specific history retention. Sources checked 28 October 2026. The workshop, code, counter and receiving exercise are original LockedIn Labs materials; no vendor implementation or partnership is claimed.