Free learning / 50-minute HTTP and reconciliation lab
Recover HTTP work when the response disappears
Run real loopback HTTP disconnects and an in-flight timeout. Reconcile matching receipts, challenge an early 404 and expose duplicate effects from a changed replay key.
No account needed. Use your own copy for writing and keep the original as a reference.
Engineering lab library / Run locally
Choose your next agent lab.
Start with the durable workflow reference, then choose the boundary you want to test. Open its walkthrough for setup, commands and expected observations.
Synthetic records. No provider key or model inference. Runs locally with the standard library.
Start here / Python 3.11+ · Standard library
HTTP outcomes and receipt reconciliation
Observe disconnects on both sides of commit, a late effect after timeout, stable replay and a mismatched new-key receipt.
Python 3.11+ · Standard library · Owned loopback listener · Use --http
Original LockedIn Labs workshop · 50 minutes. Use Python 3.11+ and its standard library. The downloaded runner opens an owned listener on 127.0.0.1 and a temporary operating-system-selected port. Two temporary SQLite databases hold fictional caller intent and receiver rows. No packages, Docker, cloud, account, credentials or external host are needed. Read the file before running it.
You will distinguish a transport observation from a committed effect, reproduce a timeout while work remains in flight, and reconcile a returned receipt against the original operation. Keep the transactional-outbox exercise beside this lesson. That original child-process crash remains a separate mode; this workshop uses actual HTTP requests and socket closure.
1. Name what an acknowledgement proves
A request asks a service to do work. A response is the HTTP message returned to the client. An effect is the state change this exercise commits in the receiver database. A receipt is the receiver's stored application record binding an operation to its payload and effect. It is not a TCP acknowledgement or proof of every downstream action.
An unknown outcome means the caller lacks enough evidence to say whether its operation completed. A timeout limits the caller's waiting. A disconnected socket says the response was not received. Neither observation alone establishes that the database rolled back.
Idempotent operation semantics let a supported repeated request preserve the intended effect. POST does not supply those semantics merely by being HTTP. This receiver adds a specific contract: a supplied scope plus operation key identifies one canonical payload and stored result; effect and receipt commit in the same transaction. Changing the key asks for a different operation. Retention, authorization and other side effects remain separate design questions.
Illustrative learning artifact
One event. Two transaction boundaries.
event-42 → delivery may repeat →
2. Follow the request and commit separately
The caller first commits the fictional OP-1 intent and its pending outbox row. A loopback HTTP handler then receives the request. The fixture closes one connection before applying the effect, and another after effect and receipt commit. Both clients observe RemoteDisconnected. Only the independent exercise observer can compare those database states; its private read does not acknowledge the caller's work.
A third case holds the received request at a finite event gate before its effect. The client actually times out. A concurrent GET cannot find a receipt yet. The fixture then releases the handler, which commits despite the client's timeout and closes its response path. That earlier 404 was an observation while work remained in flight, not a durable promise that nothing would happen.
The following visual explains original traces from the runner. It does not open a service in the browser, execute a database or assess a learner.
UNDERSTAND THE SYSTEM
The response can disappear on either side of a commit
Follow actual loopback HTTP observations and separate SQLite commits. These authored frames compare independently reset cases and a changed replay key; they do not run a server in the browser.
Watch one example, step by step.
Read the full worked example
- 011 · One disconnect, two possible effects
The before-effect and after-commit cases both raise RemoteDisconnected with no HTTP response. Independent database observations differ: zero versus one effect. Those private observer counts do not close the caller's pending item.
ACTUAL TRANSPORT OBSERVATION: Same exception · observer effects 0 or 1
- 022 · A timeout does not release the work
The handler has received the gated request before the client times out. A concurrent GET returns 404 while the handler still waits before its effect. The missing receipt is real but cannot promise that the request will never commit.
ACTUAL IN-FLIGHT LOOKUP: TimeoutError · receipt 404 · handler unfinished
- 033 · The late effect can still commit
The fixture releases the handler after the timeout and early lookup. Effect and OP-1 receipt commit; the response path is closed. The caller remains unknown until it receives matching evidence. Its timeout did not cancel this database operation.
ACTUAL POST-TIMEOUT COMMIT: Earlier 404 · later committed effect 1
- 044 · Same identity recovers the stored result
One explicit OP-1 replay applies once if nothing happened, or returns replayed with the stored receipt after a commit. The response matches the original scope, key and digest, so the caller can acknowledge it. All three default cases retain one effect.
SAME-KEY RESULT: 201 applied or 200 replayed · final effect count 1
- 055 · A new key creates different work
Change only --retry-key to OP-2. After a committed first request, the replay returns HTTP 201 for another effect. Its receipt cannot close the original OP-1 outbox item. The caller remains unknown after that successful but mismatched response.
CHANGED REPLAY IDENTITY: OP-1 intent · OP-2 receipt · two effects after original commit
- 066 · Confirmation does not undo a duplicate
A final original-key GET can return the stored OP-1 receipt and reconcile that operation. The extra OP-2 effect remains. In the before-effect case there is no original OP-1 receipt, so it stays unknown. Keep identity reconciliation and duplicate-effect repair separate for the receiver.
READ-ONLY RECONCILIATION: Original can be confirmed while two effects remain
Read the connections in plain English
- 01Original caller → HTTP observation
The client sends the original intent without an automatic retry.
- 02HTTP observation → Receiver operation
Losing the response does not establish whether the handler completed.
- 03Receiver operation → Effect + receipt
The effect and scoped receipt commit atomically in this receiver.
- 04Effect + receipt → Receipt lookup
The lookup returns actual stored evidence for a supplied fixture identity.
- 05Receipt lookup → Original caller
The original scope, key and payload digest must match before acknowledgment.
- 06Original caller → Receiving review
The caller preserves uncertainty when no matching result was received.
- 07Receipt lookup → Receiving review
Reconciliation and any duplicate-effect repair remain distinct decisions.
Check your understanding: Does a 404 receipt lookup after a timeout prove that no effect will happen?
No. In the gated case, the lookup returns 404 while the handler is still in flight; it then commits after the client timed out. Use the operation's defined reconciliation contract and retain uncertainty until matching evidence arrives. A new key can add duplicate work.
3. Run the actual HTTP faults
Download the current outbox and HTTP runner. Older copies might not include --http.
python3 --version
python3 fde-outbox-replay.py --http > http-original.jsonThe explicit mode runs three independently reset cases:
| Case | Authored fault | What the caller initially observes |
|---|---|---|
drop-before-effect | Handler closes the socket before the receiver operation | No HTTP response; outcome unknown |
drop-after-commit | Handler commits effect and receipt, then closes before replying | No HTTP response; outcome unknown |
timeout-in-flight | Received request waits at an event gate past the client's timeout | Actual timeout; receipt lookup is initially 404 |
The client uses http.client with no automatic POST retry. One later POST replay is explicit. The 0.4-second client timeout, finite gate and socket bounds coordinate this small teaching fixture; they are not measured service latency or recommended production settings. The gate uses events rather than an assumed sleep to establish which request is still in flight.
The listener accepts only the authored routes and synthetic payload. Scope labels north and south are supplied partitions, not authenticated callers. This HTTP server has no TLS or real access policy and is not a production service template. It binds loopback only, accepts no arbitrary hostname or database path, closes its owned listener, and removes its temporary files. If your environment prohibits local listeners, keep the reading exercise labeled as a prediction instead of claiming execution.
Retain the Python/SQLite versions, original delivery, actual exception or HTTP status/body, caller state, observer-only counts, server trace and confirmed listener closure. The response field is null after a transport exception; the runner does not invent an HTTP 500 or 504 status for it.
4. Inspect recovery with the same key
With the default OP-1 replay:
- The before-effect case applies the operation once and returns HTTP 201 with a stored receipt.
- The after-commit and in-flight-timeout cases return HTTP 200 with
replayedand the existing receipt. Both retain one effect. - Only the returned receipt matching the original scope, operation key and payload digest lets the caller mark its original outbox row acknowledged.
In timeout-in-flight, locate early_lookup_while_in_flight: its HTTP status is 404. In server_trace, locate the later receiver_result for the first request. The operation committed after the client had stopped waiting. Compare this with independent_observer_after_fault and the eventual matching result. Do not describe timeout as cancellation.
A missing receipt in a real integration may reflect in-flight work, visibility delay, expiry, the wrong identity or an unperformed operation. This fixture executes the in-flight explanation; it does not emulate all the others. A supported reconciliation procedure must define the lookup's meaning and the next owner when it cannot settle the outcome.
5. Change the key without changing the intent
Keep the original report, predict all three cases, then run:
python3 fde-outbox-replay.py --http --retry-key OP-2 > http-new-key.jsonThe durable caller intent stays OP-1; only the explicit replay identity becomes OP-2. The receiver accepts it as different work, even though its payload is unchanged. HTTP 201 is a successful response for that different operation. It cannot acknowledge the original intent; retry_receipt_error records receipt_intent_mismatch and caller_after_retry remains unknown.
After a committed first operation, the new key creates a second effect. The final read-only lookup of the original OP-1 receipt can reconcile that original operation, but the duplicate effect still exists. Confirmation is not compensation. The receiver/workflow owner must decide how an extra real-world effect would be investigated or repaired; this program performs no remote dispatch or compensating action.
In the before-effect case, the new key creates one effect but still leaves no OP-1 receipt. Its original caller remains unknown after lookup. Counting one receiver row does not establish that the original operation was acknowledged.
Explain to a partner: Why do the same disconnect exception and different database outcomes coexist? Why does an early 404 not prove rollback? What must match before a receipt closes an outbox item? Why does reusing a key differ from creating a new one? How can reconciliation succeed while duplicate work still needs an owner?
6. Receive the unresolved effects
Each participant keeps predictions before execution and their own original/changed reports. Give a receiver one case without interpreting it for them. Ask them to locate the transport observation, original identity, committed effect and matching-or-mismatched returned receipt. Let them reproduce the changed key. Record actual help and mark solo review as solo.
Runner version / Python / SQLite / owned listener closure:
My prediction before running / original scope, key and payload digest:
Actual exception or HTTP status / body actually received:
Caller outcome after fault / observer-only state kept separate:
Early lookup status / evidence the request was still in flight:
Original or changed replay key / returned receipt / match result:
Caller after retry / final original-key lookup / remaining effect count:
Receiving action actually attempted / assistance:
My contribution / unresolved effect / next owner and bounded action:The application engineer owns attempt budgets and stable operation identity. The receiver team owns its atomic effect/receipt contract, visibility and retention. The workflow owner decides what an extra effect means and how to repair it. The platform owner supplies actual transport, authentication and deployment evidence. The pod lead coordinates those decisions without acquiring their authority.
This is local evidence about fictional rows, not a claim of end-to-end exactly-once delivery, useful service throughput, operational independence, qualification or client placement. Carry the unresolved contract and owner into the next receiving task.
7. Check the primary references
Sources checked 27 October 2026: RFC 9110, idempotent methods, AWS Builders' Library, making retries safe with idempotent APIs and Python HTTP server documentation. These inform the boundary; the fixture, commands, visual and receiving exercise are original LockedIn Labs teaching material.