# Recover HTTP work when the response disappears

Original LockedIn Labs workshop · 50 minutes. Use Python 3.11+ and its standard library. The downloaded runner opens an owned listener on `127.0.0.1` and a temporary operating-system-selected port. Two temporary SQLite databases hold fictional caller intent and receiver rows. No packages, Docker, cloud, account, credentials or external host are needed. Read the file before running it.

You will distinguish a transport observation from a committed effect, reproduce a timeout while work remains in flight, and reconcile a returned receipt against the original operation. Keep the [transactional-outbox exercise](https://training.lockedinlabs.ai/training/guides/action-recovery) beside this lesson. That original child-process crash remains a separate mode; this workshop uses actual HTTP requests and socket closure.

## 1. Name what an acknowledgement proves

A **request** asks a service to do work. A **response** is the HTTP message returned to the client. An **effect** is the state change this exercise commits in the receiver database. A **receipt** is the receiver's stored application record binding an operation to its payload and effect. It is not a TCP acknowledgement or proof of every downstream action.

An **unknown outcome** means the caller lacks enough evidence to say whether its operation completed. A timeout limits the caller's waiting. A disconnected socket says the response was not received. Neither observation alone establishes that the database rolled back.

**Idempotent operation semantics** let a supported repeated request preserve the intended effect. POST does not supply those semantics merely by being HTTP. This receiver adds a specific contract: a supplied scope plus operation key identifies one canonical payload and stored result; effect and receipt commit in the same transaction. Changing the key asks for a different operation. Retention, authorization and other side effects remain separate design questions.

## 2. Follow the request and commit separately

The caller first commits the fictional `OP-1` intent and its pending outbox row. A loopback HTTP handler then receives the request. The fixture closes one connection before applying the effect, and another after effect and receipt commit. Both clients observe `RemoteDisconnected`. Only the independent exercise observer can compare those database states; its private read does not acknowledge the caller's work.

A third case holds the received request at a finite event gate before its effect. The client actually times out. A concurrent GET cannot find a receipt yet. The fixture then releases the handler, which commits despite the client's timeout and closes its response path. That earlier 404 was an observation while work remained in flight, not a durable promise that nothing would happen.

The following visual explains original traces from the runner. It does not open a service in the browser, execute a database or assess a learner.

## 3. Run the actual HTTP faults

Download the current [outbox and HTTP runner](https://training.lockedinlabs.ai/templates/fde-outbox-replay.py). Older copies might not include `--http`.

```sh
python3 --version
python3 fde-outbox-replay.py --http > http-original.json
```

The explicit mode runs three independently reset cases:

| Case | Authored fault | What the caller initially observes |
| --- | --- | --- |
| `drop-before-effect` | Handler closes the socket before the receiver operation | No HTTP response; outcome unknown |
| `drop-after-commit` | Handler commits effect and receipt, then closes before replying | No HTTP response; outcome unknown |
| `timeout-in-flight` | Received request waits at an event gate past the client's timeout | Actual timeout; receipt lookup is initially 404 |

The client uses `http.client` with no automatic POST retry. One later POST replay is explicit. The 0.4-second client timeout, finite gate and socket bounds coordinate this small teaching fixture; they are not measured service latency or recommended production settings. The gate uses events rather than an assumed sleep to establish which request is still in flight.

The listener accepts only the authored routes and synthetic payload. Scope labels `north` and `south` are supplied partitions, not authenticated callers. This HTTP server has no TLS or real access policy and is not a production service template. It binds loopback only, accepts no arbitrary hostname or database path, closes its owned listener, and removes its temporary files. If your environment prohibits local listeners, keep the reading exercise labeled as a prediction instead of claiming execution.

Retain the Python/SQLite versions, original delivery, actual exception or HTTP status/body, caller state, observer-only counts, server trace and confirmed listener closure. The response field is null after a transport exception; the runner does not invent an HTTP 500 or 504 status for it.

## 4. Inspect recovery with the same key

With the default `OP-1` replay:

- The before-effect case applies the operation once and returns HTTP 201 with a stored receipt.
- The after-commit and in-flight-timeout cases return HTTP 200 with `replayed` and the existing receipt. Both retain one effect.
- Only the returned receipt matching the original scope, operation key and payload digest lets the caller mark its original outbox row acknowledged.

In `timeout-in-flight`, locate `early_lookup_while_in_flight`: its HTTP status is 404. In `server_trace`, locate the later `receiver_result` for the first request. The operation committed after the client had stopped waiting. Compare this with `independent_observer_after_fault` and the eventual matching result. Do not describe timeout as cancellation.

A missing receipt in a real integration may reflect in-flight work, visibility delay, expiry, the wrong identity or an unperformed operation. This fixture executes the in-flight explanation; it does not emulate all the others. A supported reconciliation procedure must define the lookup's meaning and the next owner when it cannot settle the outcome.

## 5. Change the key without changing the intent

Keep the original report, predict all three cases, then run:

```sh
python3 fde-outbox-replay.py --http --retry-key OP-2 > http-new-key.json
```

The durable caller intent stays `OP-1`; only the explicit replay identity becomes `OP-2`. The receiver accepts it as different work, even though its payload is unchanged. HTTP 201 is a successful response for that different operation. It cannot acknowledge the original intent; `retry_receipt_error` records `receipt_intent_mismatch` and `caller_after_retry` remains unknown.

After a committed first operation, the new key creates a second effect. The final read-only lookup of the original `OP-1` receipt can reconcile that original operation, but the duplicate effect still exists. Confirmation is not compensation. The receiver/workflow owner must decide how an extra real-world effect would be investigated or repaired; this program performs no remote dispatch or compensating action.

In the before-effect case, the new key creates one effect but still leaves no `OP-1` receipt. Its original caller remains unknown after lookup. Counting one receiver row does not establish that the original operation was acknowledged.

Explain to a partner: Why do the same disconnect exception and different database outcomes coexist? Why does an early 404 not prove rollback? What must match before a receipt closes an outbox item? Why does reusing a key differ from creating a new one? How can reconciliation succeed while duplicate work still needs an owner?

## 6. Receive the unresolved effects

Each participant keeps predictions before execution and their own original/changed reports. Give a receiver one case without interpreting it for them. Ask them to locate the transport observation, original identity, committed effect and matching-or-mismatched returned receipt. Let them reproduce the changed key. Record actual help and mark solo review as solo.

```text
Runner version / Python / SQLite / owned listener closure:
My prediction before running / original scope, key and payload digest:
Actual exception or HTTP status / body actually received:
Caller outcome after fault / observer-only state kept separate:
Early lookup status / evidence the request was still in flight:
Original or changed replay key / returned receipt / match result:
Caller after retry / final original-key lookup / remaining effect count:
Receiving action actually attempted / assistance:
My contribution / unresolved effect / next owner and bounded action:
```

The application engineer owns attempt budgets and stable operation identity. The receiver team owns its atomic effect/receipt contract, visibility and retention. The workflow owner decides what an extra effect means and how to repair it. The platform owner supplies actual transport, authentication and deployment evidence. The pod lead coordinates those decisions without acquiring their authority.

This is local evidence about fictional rows, not a claim of end-to-end exactly-once delivery, useful service throughput, operational independence, qualification or client placement. Carry the unresolved contract and owner into the next receiving task.

## 7. Check the primary references

Sources checked 27 October 2026: [RFC 9110, idempotent methods](https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2), [AWS Builders' Library, making retries safe with idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) and [Python HTTP server documentation](https://docs.python.org/3.11/library/http.server.html). These inform the boundary; the fixture, commands, visual and receiving exercise are original LockedIn Labs teaching material.