# Worker fencing: a lost lease does not undo the effect

Original LockedIn Labs proof workshop · 50 planned minutes, excluding setup · Python 3.11+ standard library. Two owned local processes use two temporary SQLite files. The runner makes actual database changes and pauses an actual worker; the clock is injected logical time. There is no network, broker, model, real tool action or production authentication.

## 1. Separate ownership from the work already done

A **lease** gives a worker a bounded ownership interval under an authority's rules. Expiry lets another worker take over; it does not necessarily stop the old process. A paused worker can resume, or an earlier request can arrive late.

A **fencing token** identifies an ownership generation. In this exercise, the queue issues increasing integers. The destination keeps the highest token committed by an admitted, non-conflicting transaction for its one synthetic resource and rejects smaller tokens. A received token or an uncommitted update does not establish that stored boundary. The token is not an access credential, a timestamp, a business operation ID or proof that the sender is currently authorized.

An **operation receipt** answers another question: did this destination already apply this particular business operation with this payload? Equal fencing tokens are allowed, so fencing alone does not suppress a same-owner retry. Stable operation identity, payload agreement and an atomic receipt/effect boundary are separate controls.

The existing worker-recovery lab protects completion inside its queue database. This workshop tests the next boundary: what happens if the worker changes a separate destination before discovering that it lost the queue lease?

## 2. Follow two databases through one pause

Worker A claims `JOB-1` at logical tick 0, with fence 1 and expiry 5. It reads the job successfully and pauses **before** calling the destination. After receiving that real pause event, the parent process also acts as worker B: it claims the same job at tick 5 and receives fence 2. This is two real processes with separate connections, not a distributed lease service.

Both workers represent the same intended `OP-1`, adding four units to one fictional counter. The queue and destination are separate SQLite files. No transaction spans both. When A eventually calls `finish` at tick 6, the queue refuses it with `lease_lost`. Inspect the destination anyway: its earlier committed effect may remain.

The resource gate compares `supplied_fence < committed_high_water` inside its writer transaction. It advances the high-water mark only for admitted, non-conflicting work, including a matching receipt replay. Its stored mark survives closing and reopening that scratch file. It does not ask the queue whether the lease is current.

This distinction follows the destination-side sequencer problem described in section 2.4 of the original [Chubby paper](https://storage.googleapis.com/gweb-research2023-media/pubtools/4444.pdf). Our integers, clock, schema and counter are an original teaching fixture, not a Chubby implementation.

## 3. Predict and run six controlled receiving cases

Download [the standalone Python runner](/templates/fde-worker-fencing.py) into a disposable folder and read it first:

```sh
python3 --version
python3 fde-worker-fencing.py > fencing-original.json
python3 fde-worker-fencing.py --case fence-old-first
python3 fde-worker-fencing.py --case fence-and-receipt --units 3 > fencing-three.json
```

On Windows, use `py -3` if that is your Python launcher. There are no package installations or outside requests. Each case owns and cleans up its child process and temporary databases. Do not reuse an output filename if you want to retain the earlier report. The runner accepts only a named case and units from 1 through 9; it accepts no secret, service URL or arbitrary database path.

Predict the six baseline outcomes before inspecting the report. The seventh rollback comparison is explained in the next section:

| Case | Destination contract and delivery order | Actual final units / receipts |
| --- | --- | --- |
| `finish-only` | No destination fencing; B applies, then paused A applies | 8 / 0; A's queue completion is still refused |
| `fence-new-first` | B establishes highest 2 before A arrives | 4 / 0; A is fenced at the destination |
| `fence-old-first` | A arrives after queue reclaim but before destination sees 2 | 8 / 0; both destination calls apply |
| `same-fence-repeat` | B repeats fence 2; A later arrives | 8 / 0; equal tokens do not deduplicate |
| `fence-and-receipt` | A arrives first; B and its repeat use the same operation/units | 4 / 1; B replays and advances highest to 2 |
| `changed-operation` | B applies, then retries OP-1 with changed units | 4 / 1; the changed payload conflicts |

All seven default queue histories finish with worker B, fence 2 and two claims. That identical queue state hides very different destination effects. `attempts` retains the actual before/after resource state and destination result in the controlled arrival order. The old worker's process ID differs from the coordinator/new worker's process ID; `old_worker_exit` records a normal exit. This is a pause/resume exercise, not a worker-kill or network-partition experiment.

The second command isolates the subtle case: A's effect can be admitted at destination highest 0 even though the queue already assigned fence 2. The third changes the operation from four to three units; it should retain three units and one receipt, not an invented four-unit reference.

## 3a. Observe the fence rollback boundary

The default report now includes a seventh comparison, `fence-rollback`. Run it separately before assuming that receiving a newer token establishes a durable fence:

```sh
python3 fde-worker-fencing.py --case fence-rollback > fencing-rollback.json
python3 fde-worker-fencing.py --case fence-rollback --units 3 > fencing-rollback-three.json
```

B holds the current queue lease at fence 2. Its destination transaction updates the high-water mark to 2 and increments the counter, then a deliberate exception occurs **before** the receipt insert and commit. `uncommitted_failure_view` records only that failing connection's own view. It is not a committed snapshot available to worker A. The transaction rolls back before A is released.

| Actual attempt | Failing connection's temporary view | Committed destination after the attempt |
| --- | --- | --- |
| B / fence 2 / controlled failure | highest 2, four units, no receipt | highest 0, zero units, no receipt |
| A / fence 1 / released after rollback | not applicable | highest 1, four units, one receipt |
| B / fence 2 / matching retry | not applicable | highest 2, four units, one receipt; replayed |

The first attempt’s `queue_after_destination` records B still running at fence 2 after the destination rollback. A's destination call is admitted against committed mark 0 even though the destination previously attempted to process 2. A's queue completion is refused. B's later matching replay commits mark 2 without another increment. With `--units 3`, the temporary and retained effect values are three instead of four; the zero-after-rollback observation remains zero.

The fixture’s `highest_fence` is therefore a **committed admission mark**, not a log of every token received. This is the contract implemented by this atomic destination transaction, not a universal fencing-service specification. A design that persists an epoch independently from the business action has another recovery boundary and needs separate tests. This controlled exception is not an OS crash, power loss or unavailable database.

## 4. Keep fencing, freshness and idempotency distinct

This destination’s committed fence protects an ordering boundary: after an admitted transaction establishes 2, a request carrying 1 cannot apply there. A failed or conflicting transaction rolls its tentative high-water update back with its effect. It does **not** establish immediate revocation when the issuer grants 2, and it does not prove wall-clock expiry. A separate current-authority contract may be needed; an early worker-side check followed by a remote effect leaves a gap.

A same-fence repeat remains admissible. In `same-fence-repeat`, both B calls add four units. In the combined candidate, the destination instead binds `OP-1` to the requested units. A matching retry returns the stored result without another increment. A different units value conflicts and rolls back the transaction. The high-water mark, receipt and effect live in one local transaction; they do not share a transaction with the queue.

The combined old-first case is deliberately limited: A still performs the first local effect while its queue lease is stale. The matching operation receipt prevents a second effect. It does not retroactively authorize A, choose a correct changed payload, authenticate either worker or undo an already committed action. Real permissions, operation meaning and recovery remain distinct.

The runner uses explicit `BEGIN IMMEDIATE`, followed by commit or rollback, on connections with `isolation_level=None`. Read the primary [SQLite transaction contract](https://sqlite.org/lang_transaction.html) and [Python transaction-control reference](https://docs.python.org/3.11/library/sqlite3.html#transaction-control) before adapting the pattern. SQLite serializes its writers; a local counter result is not evidence of remote exactly-once execution.

## 5. Hand the proof to another engineer

Spend the final ten minutes on a receiving task. One person retains the original report; another predicts `--case fence-old-first --units 3`, runs it themselves and locates the two actual effects totaling six. Swap roles for `fence-and-receipt --units 3`: the receiver must locate one three-unit effect, one receipt and highest fence 2. Keep each person's prediction, command, observed evidence, assistance and correction separate. Solo practice is self-review.

For the rollback case, have a receiver locate the failing connection’s temporary mark, the zero committed mark after rollback and the unchanged queue lease before explaining A’s admission. Keep those three observations distinct.

The application owner names the stable operation, payload agreement and reconciliation rule. The queue/platform owner explains token issuance, scope, persistence, clock and recovery after failover or restore. The destination owner must show where the token is checked and where the receipt/effect commits. The pod lead records which boundary remains untested and who receives the next bounded task. A shared presentation or a green process exit does not establish an individual's independence or qualification.

Before a real adaptation, agree the token's issuer and resource scope, authenticated transport, monotonicity across recovery, current-authority expectations, receipt retention, payload conflicts, concurrent requests and uncertain destination outcomes. Do not assume two independent locks' token numbers are comparable or that a token alone authorizes access. This fixture supplies trusted inputs for one job and one resource; it implements none of that production infrastructure.

## 6. Defend the decision in an interview

- **Why did A's refused completion fail to protect the destination?** The destination committed first in another database. A queue update cannot undo that effect.
- **Why did the old-first fenced case still apply twice?** The destination had not yet observed the newer token, and fencing did not deduplicate OP-1.
- **Why does receiving token 2 not always establish the stored fence?** In this fixture the mark commits with the destination transaction. A rollback preserves the earlier committed mark, even when the failed connection briefly saw 2.
- **Why is the combined case still not a complete authority guarantee?** A performed the first effect after losing its queue lease. A receipt limits repetition; it does not prove current permission or a correct business payload.

Keep a short record: original prediction, exact runtime/file/command, each destination state, the separate queue outcome, one changed-input comparison, help received and the next accountable owner. Continue with [uncertain action recovery](/training/guides/action-recovery) to investigate destination receipts when the caller does not know whether an effect committed.