Worker fencing: a lost lease does not undo the effect

A lease gives a worker a bounded ownership interval under an authority's rules. Expiry lets another worker take over; it does not necessarily stop the old process. A paused worker can resume, or an earlier request can arrive late.

A fencing token identifies an ownership generation. In this exercise, the queue issues increasing integers. The destination keeps the highest token committed by an admitted, non-conflicting transaction for its one synthetic resource and rejects smaller tokens. A received token or an uncommitted update does not establish that stored boundary. The token is not an access credential, a timestamp, a business operation ID or proof that the sender is currently authorized.

An operation receipt answers another question: did this destination already apply this particular business operation with this payload? Equal fencing tokens are allowed, so fencing alone does not suppress a same-owner retry. Stable operation identity, payload agreement and an atomic receipt/effect boundary are separate controls.

The existing worker-recovery lab protects completion inside its queue database. This workshop tests the next boundary: what happens if the worker changes a separate destination before discovering that it lost the queue lease?

Worker A claims JOB-1 at logical tick 0, with fence 1 and expiry 5. It reads the job successfully and pauses before calling the destination. After receiving that real pause event, the parent process also acts as worker B: it claims the same job at tick 5 and receives fence 2. This is two real processes with separate connections, not a distributed lease service.

Both workers represent the same intended OP-1, adding four units to one fictional counter. The queue and destination are separate SQLite files. No transaction spans both. When A eventually calls finish at tick 6, the queue refuses it with lease_lost. Inspect the destination anyway: its earlier committed effect may remain.

The resource gate compares supplied_fence < committed_high_water inside its writer transaction. It advances the high-water mark only for admitted, non-conflicting work, including a matching receipt replay. Its stored mark survives closing and reopening that scratch file. It does not ask the queue whether the lease is current.

This distinction follows the destination-side sequencer problem described in section 2.4 of the original Chubby paper. Our integers, clock, schema and counter are an original teaching fixture, not a Chubby implementation.

Download the standalone Python runner into a disposable folder and read it first:

python3 --version
python3 fde-worker-fencing.py > fencing-original.json
python3 fde-worker-fencing.py --case fence-old-first
python3 fde-worker-fencing.py --case fence-and-receipt --units 3 > fencing-three.json

On Windows, use py -3 if that is your Python launcher. There are no package installations or outside requests. Each case owns and cleans up its child process and temporary databases. Do not reuse an output filename if you want to retain the earlier report. The runner accepts only a named case and units from 1 through 9; it accepts no secret, service URL or arbitrary database path.

Predict the six baseline outcomes before inspecting the report. The seventh rollback comparison is explained in the next section:

All seven default queue histories finish with worker B, fence 2 and two claims. That identical queue state hides very different destination effects. attempts retains the actual before/after resource state and destination result in the controlled arrival order. The old worker's process ID differs from the coordinator/new worker's process ID; old_worker_exit records a normal exit. This is a pause/resume exercise, not a worker-kill or network-partition experiment.

The second command isolates the subtle case: A's effect can be admitted at destination highest 0 even though the queue already assigned fence 2. The third changes the operation from four to three units; it should retain three units and one receipt, not an invented four-unit reference.

The default report now includes a seventh comparison, fence-rollback. Run it separately before assuming that receiving a newer token establishes a durable fence:

python3 fde-worker-fencing.py --case fence-rollback > fencing-rollback.json
python3 fde-worker-fencing.py --case fence-rollback --units 3 > fencing-rollback-three.json

B holds the current queue lease at fence 2. Its destination transaction updates the high-water mark to 2 and increments the counter, then a deliberate exception occurs before the receipt insert and commit. uncommitted_failure_view records only that failing connection's own view. It is not a committed snapshot available to worker A. The transaction rolls back before A is released.

The first attempt’s queue_after_destination records B still running at fence 2 after the destination rollback. A's destination call is admitted against committed mark 0 even though the destination previously attempted to process 2. A's queue completion is refused. B's later matching replay commits mark 2 without another increment. With --units 3, the temporary and retained effect values are three instead of four; the zero-after-rollback observation remains zero.

The fixture’s highest_fence is therefore a committed admission mark, not a log of every token received. This is the contract implemented by this atomic destination transaction, not a universal fencing-service specification. A design that persists an epoch independently from the business action has another recovery boundary and needs separate tests. This controlled exception is not an OS crash, power loss or unavailable database.

This destination’s committed fence protects an ordering boundary: after an admitted transaction establishes 2, a request carrying 1 cannot apply there. A failed or conflicting transaction rolls its tentative high-water update back with its effect. It does not establish immediate revocation when the issuer grants 2, and it does not prove wall-clock expiry. A separate current-authority contract may be needed; an early worker-side check followed by a remote effect leaves a gap.

A same-fence repeat remains admissible. In same-fence-repeat, both B calls add four units. In the combined candidate, the destination instead binds OP-1 to the requested units. A matching retry returns the stored result without another increment. A different units value conflicts and rolls back the transaction. The high-water mark, receipt and effect live in one local transaction; they do not share a transaction with the queue.

The combined old-first case is deliberately limited: A still performs the first local effect while its queue lease is stale. The matching operation receipt prevents a second effect. It does not retroactively authorize A, choose a correct changed payload, authenticate either worker or undo an already committed action. Real permissions, operation meaning and recovery remain distinct.

The runner uses explicit BEGIN IMMEDIATE, followed by commit or rollback, on connections with isolation_level=None. Read the primary SQLite transaction contract and Python transaction-control reference before adapting the pattern. SQLite serializes its writers; a local counter result is not evidence of remote exactly-once execution.

Spend the final ten minutes on a receiving task. One person retains the original report; another predicts --case fence-old-first --units 3, runs it themselves and locates the two actual effects totaling six. Swap roles for fence-and-receipt --units 3: the receiver must locate one three-unit effect, one receipt and highest fence 2. Keep each person's prediction, command, observed evidence, assistance and correction separate. Solo practice is self-review.

For the rollback case, have a receiver locate the failing connection’s temporary mark, the zero committed mark after rollback and the unchanged queue lease before explaining A’s admission. Keep those three observations distinct.

The application owner names the stable operation, payload agreement and reconciliation rule. The queue/platform owner explains token issuance, scope, persistence, clock and recovery after failover or restore. The destination owner must show where the token is checked and where the receipt/effect commits. The pod lead records which boundary remains untested and who receives the next bounded task. A shared presentation or a green process exit does not establish an individual's independence or qualification.

Before a real adaptation, agree the token's issuer and resource scope, authenticated transport, monotonicity across recovery, current-authority expectations, receipt retention, payload conflicts, concurrent requests and uncertain destination outcomes. Do not assume two independent locks' token numbers are comparable or that a token alone authorizes access. This fixture supplies trusted inputs for one job and one resource; it implements none of that production infrastructure.

• Why did A's refused completion fail to protect the destination? The destination committed first in another database. A queue update cannot undo that effect.
• Why did the old-first fenced case still apply twice? The destination had not yet observed the newer token, and fencing did not deduplicate OP-1.
• Why does receiving token 2 not always establish the stored fence? In this fixture the mark commits with the destination transaction. A rollback preserves the earlier committed mark, even when the failed connection briefly saw 2.
• Why is the combined case still not a complete authority guarantee? A performed the first effect after losing its queue lease. A receipt limits repetition; it does not prove current permission or a correct business payload.

Keep a short record: original prediction, exact runtime/file/command, each destination state, the separate queue outcome, one changed-input comparison, help received and the next accountable owner. Continue with uncertain action recovery to investigate destination receipts when the caller does not know whether an effect committed.