Why Expired Task Leases Still Allow Writes: Execution Epochs, Takeover Barriers and Failure Tests

Lease expiry permits reassignment but does not stop the previous worker. A PostgreSQL 18 design and a report corrected from 1000 to 900 explain atomic checks of execution epochs, approved inputs and operation receipts, compare database-local coordination with external locks, and test delayed writes, renewal races, unknown results and recovery.

Preventing an expired worker from overwriting a newer result requires the final write service to reject disqualified execution epochs and make that check atomic with the mutation. A longer lease, periodic heartbeats or one last lock lookup cannot close the delayed-write window alone. For scheduling, data pipelines and agent tool services, this article develops a verifiable takeover protocol around publishing one report resource.

Define the protected resource and the exact invariants

A lease grants time-limited ownership. A worker may continue a task while eligible, and another can take over after loss of contact or expiry. Expiry changes the service's assignment state; it does not forcibly terminate the old process. Garbage-collection pauses, frozen hosts and network delays can all let an old worker send a request after a new holder has started.[1]

An execution epoch, often called a fencing token, advances on each valid takeover. Bind it to the protected resource, issue it through a trusted service and carry it to the final receiver on every write. A random owner identifier distinguishes holders but does not establish order. A client timestamp cannot substitute for ordered server-side issuance.

The resource here is company A's monthly report R. Computation may take time, while the public pointer and committed result receipts reside in one PostgreSQL 18 database. Every publication entry point must use the same service; workers cannot overwrite result tables directly. External payments, equipment actions and atomic commits across databases are outside this local guarantee.

Three invariants define the design: after a newer epoch's takeover transaction commits, an older epoch cannot change R; one operation key identifies one definite result, so retries cannot create a second effect; and a published result must use the currently approved input version. A valid epoch does not establish correct computation, and a valid input does not establish current ownership. All three checks serve different purposes.

Define expiry precisely as well. After acquiring the database row lock, this design reads current server time to decide whether to authorize the short transaction. It does not promise to physically revoke an authorized transaction at the exact instant a real-time deadline passes. A takeover must acquire the same row lock, giving authorization, commit and takeover a defined order.

Reproduce stale overwriting with a paused-worker timeline

Construct two report inputs. In immutable snapshot S1, order O1 is 600 and O2 is 400, totaling 1000. An approved correction creates S2, retaining O1 at 600 and changing O2 to 300, totaling 900. Both use the same monetary definition. Worker A holds an S1 task; worker B receives the approved S2 task. They are different frozen inputs and must remain distinguishable.

At relative time zero, A acquires epoch 41 for R with expiry at thirty seconds. At second five it begins a long pause. At second thirty-one, B takes over as epoch 42 and the service sets the approved input to S2. B commits 900 at second thirty-five. A resumes at second forty and submits 1000 with epoch 41 and S1. A service that overwrites by report identifier alone will regress to 1000.

The expected result is to reject A at second forty, keeping R on S2 and 900 and generating no publication event for the rejected request. Even a lock lookup immediately before writing is insufficient: A can pause again after the lookup. Eligibility checking must be bound to publication within a critical section that cannot admit a takeover between check and effect.

A stronger counterexample delays B's business publication after its takeover at second thirty-one. A reaches the write service first at second thirty-two. If that service remembers only the highest token from a successful result write, it may still know only epoch 41 and admit A. A highest-seen-token rule prevents regression after a higher token is observed; it does not inherently mean a new owner has completed takeover. Specify when the new epoch takes effect at the resource.

Here, takeover means the database transaction advancing the current epoch has committed, rather than an external lock service merely returning a number. A is therefore rejected even before B produces 900. The old public result may remain visible with an updating state. A disqualified computation should not become a temporary new result merely to keep a page looking fresh.

Separate resource identity, ownership and business operations

Provision one control row per protected resource, uniquely keyed by company and resource identifier. Store a resource generation, execution epoch, owner, lease deadline, approved input version and state. The generation distinguishes a recreated resource or a new recovery boundary from its older namesake. Within a generation, the epoch advances monotonically rather than resetting whenever a task restarts.

Result records contain company, resource, input version, result digest or immutable result address, amount and producing epoch. Operation receipts are unique by company and operation key, retaining resource identity, input digest, output digest and committed outcome. A publishing S1 and B publishing S2 are different operations. B's network retries retain its original key and content.

The example calls the operations Publish-S1 and Publish-S2, but production keys need a scope that includes company and precise business intent, not merely an easily reused short name. Changing 900 to 950 under the same key must cause a content conflict. Adding a new retry timestamp to the key instead prevents recognition of the same operation.

An epoch is not a password. The service authenticates the caller, checks resource authorization and verifies that the control record actually granted the epoch to that holder. A client cannot submit an arbitrary 999999 and acquire superior rights. Approval of the input also comes from controlled state, not a claim in the worker's request. Receipt queries require authorization too, so deduplication cannot expose another company's results.

Distinguish success, already-completed operation, stale epoch, input-version conflict, permission denial and unknown outcome. Updating zero rows is not success; a connection timeout is not definite failure. These categories let recovery return an original receipt, reacquire work, correct inputs or wait for the original transaction to become known.

Option one: make takeover and publication atomic inside the result database

The acquisition transaction locks the control row for the company and resource with SELECT FOR UPDATE, then reads its current state. Only when work is allowed and the owner has released or the lease has expired under the protocol may it advance the epoch, bind the new owner and approved input, set the deadline and commit. A second concurrent acquirer waits and checks the updated row rather than relying on an expired state read before locking.[2]

PostgreSQL 18 row-lock behavior underpins this option: conflicting modifications and locks wait, and a lock normally lasts until the transaction ends.[2] Long model calls, file generation and network work happen after acquisition commits. Holding the row throughout computation increases waiting and connection occupancy and obstructs takeover. Short database critical sections are part of the reason to use a lease-based design.

For publication, authenticate and examine the original operation receipt first. A committed receipt with the same key and content returns the previous result without modifying R; different content conflicts. If unfinished, begin a short transaction, lock the control row in a fixed order, and check generation, holder, epoch, approved input version, state and deadline. Register the operation, write the result and update the public pointer in that same transaction.

Read the time used for expiry after obtaining the control-row lock. PostgreSQL now() and CURRENT_TIMESTAMP remain fixed at transaction start, whereas clock_timestamp() reflects the clock when called.[3] A transaction waiting ten seconds for a lock cannot use its pre-wait timestamp to demonstrate that a lease is still valid. Read current server time once for this authorization decision; do not trust a worker-supplied local timestamp.

If the deadline passes after authorization but before commit, the takeover waits on that same row lock. The original transaction commits first; the subsequent takeover evaluates the state it then reads. This guarantees a serialized takeover boundary, not hard real-time termination. Bound publication transaction duration and lock waits, and check the original operation after timeouts. Clock rollback may delay reassignment and needs monitoring; epoch fencing does not cure every availability problem.

Concurrent Publish-S2 requests also need an operation uniqueness constraint and a receipt recheck inside the lock. Once the first commits 900, the second returns that result without another publication event. Receipt, result and pointer must share the transaction boundary. Recording completion separately before the result creates false success; writing the result separately before the receipt leaves a duplicate-effect window.

Specify renewal, release and multiple writes within one epoch

Renewal locks the same control row and checks company, generation, holder, epoch and eligibility under the current protocol. This design forbids reviving an epoch after expiry: the worker must reacquire. Successful renewal advances the deadline without changing the epoch. An unknown renewal response is not proof of extension; read authoritative state and follow the controlled stop or recovery policy.

Release clears only eligibility matching the current holder and epoch, preserving the increasing history. A delayed release from A after epoch 42 becomes active must not clear B's lease. Redis's official lock guidance likewise checks the owner value rather than deleting a lock unconditionally.[4] That protects release, not the separate result store against stale writes.

When one epoch permits several operations, accepting only tokens strictly greater than the last token would reject the valid holder's second step. This design checks equality with the active epoch and records a separate idempotent receipt for each logical operation. A highest-token implementation must instead explicitly define retries and ordering for equal tokens.

Do not merge execution epochs with content revisions. B may still hold epoch 42 while approved input changes from S2 to S3. Whether its old computation may publish is an input-version question. The method for conditional updates and version checks addresses that content precondition. Separate ownership and business state so a rejection can identify whether the worker must reacquire or recompute.

Cancellation should reject new publications even before lease expiry. An authorized receipt query can still return a historical committed fact, but cannot reactivate the canceled resource. Recreating R requires a new resource generation. Clearing the control table is not a recovery procedure: it can make old workers appear to belong to newly created records.

Option two: external coordination needs a resource-side takeover barrier

An external coordinator can issue leases and ordered epochs when work spans services and mature coordination infrastructure already exists. The etcd 3.6 guarantees describe strictly serializable KV operations, with revisions reflecting their order; stale-permitting reads differ from default linearizable reads.[5] An ordered number in one coordinator does not automatically provide atomic authorization in another database.

To match this article's takeover semantics, the new holder first activates its epoch in the result service, which atomically advances the resource's active eligibility. Do not announce successful resource takeover before activation. Reject delayed activation of an older epoch as well. Coordinator expiry, resource-side activation and business publication are three different events and need separate records, not just an acquired-lock message.

Keep the promise precise: rejecting old epochs after activation is not the same as blocking old writes immediately when an external lease expires. The latter requires a protocol that jointly coordinates eligibility and resource mutation, or a resource-issued short-lived authorization with an explicit failure model. A network lease lookup immediately before submission still leaves a check-to-use race.

Database-local coordination removes one cross-system takeover gap, but adds contention on hot control rows and protects only effects within its transaction. External coordination may suit a well-operated system whose target services also enforce fencing, yet adds activation, partition, recovery and multi-target consistency responsibilities. If a critical target cannot check epochs or conditional writes, adding a lock service alone cannot deliver this guarantee.

A random token, a counter and a consistency mechanism are not interchangeable. Redis explicitly cautions that long-running work should consider fencing tokens and that a live process cannot assume continuing lock ownership.[4] Judge the protocol rather than the library name: when are numbers issued, can they regress after failure, and does every target write actually check them? Verify these questions with concrete timelines.

Run six fault tests against durable evidence

Test one pauses A beyond expiry, lets B take over and publish 900, then resumes A with 1000. Expect a stale-epoch rejection with B's result, input version and publication event unchanged. Inspect the control row, receipts, final pointer and rejection reason. If overwriting occurs, repair atomic checks across every write path, then reverse worker order and repeat.

Test two lets B complete takeover without publishing a result, then immediately resumes A. Expect rejection again, proving that the barrier takes effect at takeover rather than at the first new result. Record the takeover commit and A's rejection. If the maximum token lives only in the result and was not advanced, repair activation and confirm that B can still publish normally afterward.

Test three races renewal against takeover near the deadline and delays an old release. The actual serialized order determines one valid epoch: a successful renewal first prevents takeover; a valid takeover first prevents old renewal and release from changing the new record. Inspect transaction order, deadlines and holders. Client log timestamps alone cannot establish a database-confirmed order.

Test four loses the response after B commits 900, then queries and retries the same operation key and content. Expect the same committed receipt and one business effect. The same key with 950 must conflict. Inspect digests, results and event counts. If the original transaction remains unresolved, retain unknown status and investigate rather than inventing a second operation with a new key.

Test five changes approved input to S3 or cancels R, while B submits epoch 42 and S2. Expect an input-version or state rejection: a valid epoch is not unlimited authority. Also test a wrong company, a forged large epoch and an obsolete resource generation. None may create a result. After repairing conditions or authorization, use current inputs and a legitimate identity as a positive control.

Test six creates a control-row lock wait. The transaction starts before expiry but acquires the lock afterward, with no takeover yet. Under this design, reject the expired authorization and inspect time read inside the lock rather than transaction-start time. Also test crossing the deadline while already inside the critical section: subsequent takeover must wait until that transaction finishes, matching the promised serialized boundary.

Recovery and external side effects need more than a number

Restoring an old backup may roll epochs backward while old network requests still exist. The etcd recovery guidance specifically discusses revision regression after snapshot restoration and mechanisms for revision bumps and cache invalidation.[6] Recovery therefore belongs in the epoch protocol. Those options do not automatically repair every external result service's state.

For this design, first isolate old workers and write entry points, reconcile the recovery points of control records and committed receipts, establish a new resource generation or another approved non-regression boundary, and only then reopen acquisition. Incrementing a restored counter once may still leave it below a previously granted token that remains in flight. Rehearsals must submit old-generation requests and verify rejection, followed by a successful new-generation publication.

If publication also requires a notification, commit the sending intent in the same local transaction as the result, then use the transactional outbox and consumer deduplication method for delivery. That preserves an obligation to send, not a guarantee of duplicate-free networks. An expired worker must not bypass the publication service and notify directly: a database rejection would otherwise coexist with an incorrect external message.

When an external target lacks stable operation keys, outcome queries or epoch checks, leave ambiguous effects for controlled reconciliation rather than promise exactly once. Proving that 900 is committed does not prove one email was sent, much less that a device did not execute an old command. Each side effect needs its own final acceptance point.

Deliver capacity, monitoring and rollback rules together

Lease duration trades reassignment speed against renewal load; longer is not automatically safer. With ten thousand active tasks renewing every ten seconds, an ideally even load is about one thousand renewal requests per second before retries, bursts or batching. Measure peaks, tail latency, database contention and lease failures rather than choosing duration directly from average compute time.

Monitor active resources, acquisition waits, rejected renewals, expired submissions, input conflicts, time spent with unknown outcomes and the longest publication transaction. More stale-epoch rejections may indicate worker pauses, congestion, short leases or duplicate dispatch. Do not disable the barrier merely to reduce alerts. Logs should correlate resource generation, epoch and operation key without copying complete sensitive results.

The single-row example does not make multi-resource updates atomic. Within one database, use a deterministic lock order and one transaction where feasible. Cross-service effects require additional protocols and compensation boundaries. Sharding can reduce hotspots but must not arbitrarily split a business object that needs an indivisible update. Preserve the resource definition and invariants before increasing throughput.

Before rollout, execute the six timelines with authorized data and compare expected outputs with actual durable records, then expand resource scope gradually. Rollback can stop new acquisitions, resolve already-authorized short transactions and retain their receipts. It should not restore unconditional overwriting. Database recovery, coordinator upgrades and lease-duration changes require the relevant boundary tests again.

The handoff records resource granularity, input approval, epoch issuance and activation, the point at which time is checked, all write entry points, deduplication retention, ownership of unknown outcomes, backup-recovery boundaries and evidence from the six tests. The example's 1000 and 900 can be recalculated directly; concurrency and fault behavior require records from the target environment before deployment can be declared accepted.

References

  1. Martin Kleppmann — How to do distributed locking (2016-02-08)
  2. PostgreSQL 18 — Explicit Locking, Row-Level Locks
  3. PostgreSQL 18 — Date/Time Functions and Operators, Current Date/Time
  4. Redis — Distributed Locks, release, extension and consistency caveats (accessed 2026-11-20)
  5. etcd 3.6 — API guarantees
  6. etcd 3.6 — Disaster recovery, Revision Difference and revision bump
Back to insights
鲁ICP备2024109755号-2
Drag to move. Right-click, touch and hold, or press Shift+F10 to choose a corner.