Restore the data without repeating completed work

A backup preserves an earlier database state. Restoring that state can bring back pending work that the live application later completed. An effect committed in another system does not disappear because your application returned to an older point.

Imagine a job asking another service to allocate four synthetic units. The backup says pending. Later, the destination applies four units and the application records completion. Restore the earlier application copy: its schema is valid and its read query works, but it once again says pending. A worker that treats this row as proof of unfinished business can perform the effect again.

This is separate from losing recent data. Recovery Point Objective (RPO) concerns the recoverable data window; Recovery Time Objective (RTO) concerns interruption before agreed useful service returns. Those targets also need a business acceptance boundary. Neither a readable restored file nor a fast restore establishes which external operations already occurred. This workshop measures neither target.

The application owns JOB-1, carrying business operation OP-1 and its units. A destination owns the effect rows. In the guarded comparisons, it also owns an operation receipt bound to those units. An identical retry returns the stored result; a different payload for the same operation conflicts.

The destination commits effect and receipt in one SQLite transaction. No transaction spans application and destination files. The runner creates an application backup before dispatch, executes the original destination action, records application completion, then restores the earlier backup into a new application file. The completed original remains available for comparison; the destination is retained.

The mechanism below is an authored explanation. The runner's actual rows are the evidence. Its deliberate receipt deletion represents a retained-history gap; it does not implement a timed expiry service or prescribe a universal retention period.

Download the standalone runner into a disposable folder, inspect it and run:

python3 --version
python3 fde-restore-replay.py > restore-original.json
python3 fde-restore-replay.py --case retained-receipt
python3 fde-restore-replay.py --case pruned-receipt --units 3 > restore-pruned-three.json

Use py -3 on Windows if that is your launcher. No package installation is needed. The runner accepts only a named case and original units from 1 through 9; it opens no user-selected database or URL. Each case creates and cleans up its own temporary files. Save earlier reports under distinct names.

Predict the retry and destination outcome before reading the report:

All five restore the same pending application state. All five pass the actual application read and local integrity checks. Those identical observations conceal different business outcomes. With --units 3, the duplicate cases end at six and the retained/conflict cases end at three. Changed-payload requests four in that run; it does not silently change the retained original receipt.

Compare backup_job, original_application_after_completion and restored_application_before_retry. These are actual reads: pending, completed, pending. original_application_retained shows that the runner did not overwrite its completed reference.

Compare destination_before_app_restore with destination_after_app_restore: both retain the original effect. The application restore did not reset this separate database. In pruned-receipt, inspect receipt_pruning.deleted_rows and the before/after receipt lists. The effects list is still present after deletion; no cancellation or compensation happened. The unsafe retry creates a replacement receipt storing eight units in the default run. Its one final receipt does not erase the two actual effect rows.

Then compare retry_request, retry_result, final_restored_application and final_destination. A completed restored job can coexist with two effects. A replayed result is historical evidence for the same operation, not proof of current authorization or a fresh action. A conflict leaves the original destination intact and the restored job unresolved. Do not change its ID merely to obtain a successful response.

The destination is a local counter implementation. Real APIs may define different operation scope, retention, conflict and status-query contracts. A missing receipt after history pruning cannot by itself distinguish never-executed work from old completed work. Preserve that uncertainty and require an authoritative reconciliation path before deciding to retry, compensate or accept service. This lab deliberately executes the unsafe retry so you can inspect the duplicate; it is not a recovery runbook for a real service.

Allow ten minutes for a receiving review. Each person keeps a separate prediction, exact command, output pointer, own contribution, help and correction. A second engineer changes original units to three, runs pruned-receipt and locates both effect rows totaling six. Swap roles for retained-receipt --units 3: locate one effect, one receipt and the historical replay result. Solo work is self-review.

The application owner identifies stable business operations and which restored jobs might reappear. The destination owner supplies the actual receipt/status contract, payload rules and retention evidence. The recovery/platform owner identifies backup coverage, recovery points and which dependencies were retained or restored separately. The pod lead keeps unresolved operations and receiving responsibilities visible. None inherits another owner's permissions or accepts business recovery just because the runner exits successfully.

Use this blank receiving record with synthetic aliases and permitted artifacts:

Person / role / exact runner version:
Prediction before execution / command:
Backup job / completed reference / restored job evidence:
Destination retained / receipt retained or removed / actual effect rows:
Changed input / actual retry result / assistance received:
Unresolved business operation / evidence still needed:
Application owner / destination owner / recovery owner:
Proposed next action / receiving question / human decision still needed:

For cohort practice, follow the recovery pod facilitator guide. It connects three independent receiving comparisons through role swaps, a preview-only prepared agenda and blank individual worksheets.

• Why can an integrity-checked restore repeat work? The restored application can be older than a separate committed destination effect. Storage validity says nothing about that cross-system history.
• Why preserve the operation ID across restore? It lets a destination with a surviving compatible receipt recognize the same business intent. A fresh attempt ID is not a replacement for that intent.
• Does an idempotent endpoint make every restore safe? No. The caller must preserve scope and payload, and the relevant history must remain available under the actual contract. Identity, access and reconciliation still need evidence.
• What if the receipt was pruned? Missing history is insufficient evidence that the action never happened. Locate an authoritative status or reconciliation record; keep the receiving decision unresolved until supported.
• Can we claim production RTO from this result? No. These are sequential local file operations with a synthetic destination and no service outage or distributed recovery experiment.

After this lesson, use the systems recovery workshop for the two clocks and compatibility, then the outbox recovery lab for a crash between destination commit and local completion. Keep restore, crash and receipt-retention evidence distinct.

The SQLite backup API and Python backup method define the actual copy mechanism used here. The AWS Builders' Library on idempotent APIs discusses caller request identity, atomic recording, changed intent and service-specific history retention. Sources checked 28 October 2026. The workshop, code, counter and receiving exercise are original LockedIn Labs materials; no vendor implementation or partnership is claimed.