Follow a job from accepted to finished

Think of a work-order desk: accepting an order, assigning it to someone and finishing it are different events. In software, a producer submits a message to a queue. A worker receives that message and does the work. An acknowledgement tells the queue that this delivery has been handled.

The example key is north / J-12:draft:v1. The tenant, job and action stay the same when another worker retries. The delivery receipt changes. A receipt for a delivery and a receipt proving a business effect are different records.

Follow the visual one event at a time. Before advancing, predict both the message state and the number of saved drafts. A queue cannot infer a database commit from silence. A worker can stop after committing the draft but before acknowledging.

Download the Python lab into a new working folder. You need Python 3.11 or later. It uses only the standard library.

python3 --version
python3 fde-queue-delivery-lab.py test
python3 fde-queue-delivery-lab.py demo

The program creates separate temporary SQLite databases and removes them when it finishes. Each event reads committed state through a new database connection. The queue is a small teaching model: expiry uses explicit integer ticks and the interruption is an omitted acknowledgement. No worker is killed and no vendor broker is contacted. The database writes and transaction rollbacks are real.

Make predictions first. Then inspect the final event in each case:

Notice that both normal and early-ack end with a settled message. A settled queue entry alone cannot demonstrate that the intended business action happened. Keep the trace lines that establish each conclusion.

Idempotency means repeating an operation has the same intended effect as doing it once. Here the safe path writes the effect and its receipt in one SQLite transaction. On a repeat, it checks the same tenant, operation key and payload, then reuses the recorded outcome. A changed payload under the same key is a conflict; a different action needs a deliberately different key.

Read apply() in the Python file. BEGIN IMMEDIATE starts the write transaction. The receipt and draft commit together. An exception between them rolls both back. In test_receipt_rolls_back_with_effect, the next attempt can still create the draft because no false success receipt remains.

Experiment in your own copy:

1. In run_case, change the second call to use operation="J-12:draft:v2". Predict why the safe replay now creates a second effect. Run demo.
2. Restore it. Change the second call to payload="different draft" instead. Predict and observe operation_conflict in the safe case. The refusal is intentional: it must not silently call changed work a repeat.
3. Restore the original. Explain why receive(now=4) refuses a new delivery but receive(now=5) permits one in this model. Inspect the boundary test.
4. Draw the two commits separately: the business effect, then acknowledgement. Mark where the interruption occurs and which evidence the next worker reads.

This model checks the current delivery receipt at acknowledgement time. It does not cancel an old process or fence a remote service. A real service must enforce authorization again at its write boundary and bind replay to the intended input. For an external action such as sending an email, a local receipt and the remote effect do not share a SQLite transaction. Use the destination's supported idempotency and reconciliation contract; an unknown outcome is not permission to send again blindly.

These are related delivery concepts, not interchangeable APIs. Start with the provider's documented delivery mode and client behavior before choosing retries.

SQS standard delivery can produce duplicates even during a visibility period. Pub/Sub subscriptions have their own delivery and acknowledgement settings. Service Bus also offers Receive-and-Delete, which has different loss behavior from Peek-Lock. Never translate the local lab into an exactly-once claim about a cloud service or an external business effect.

For your chosen service, record the delivery mode, acknowledgement operation, expiry behavior, bounded retry policy and the owner of repeatedly failing work. A dead-letter destination holds messages for investigation; it does not repair them. Ordering, retention, permissions and operational alerts need explicit decisions too.

Official references checked 21 October 2026:

• Amazon SQS visibility timeout.
• Google Cloud Pub/Sub subscriptions and acknowledgement.
• Azure Service Bus locks and settlement.
• SQLite transaction behavior.

Keep your four predictions, the actual traces, one changed-key experiment and a diagram showing effect before acknowledgement. Explain the result without using the phrase “the queue handles it.” Name exactly which component recognizes a repeat and what evidence it stores.

Transfer challenge: the draft is now an email sent through another company's API. Your worker times out after submitting it. State what is unknown, how you would query or reconcile the original operation, and when a human must decide. The local transaction is evidence for a local write only.

Continue into the worker recovery lab for actual process interruption, lease fencing and competing workers, or open the guided Queue 101 lesson with your existing training access.