In brief: A distributed lock cannot prove a process is still the owner after pauses, partitions or expired leases.
The problem
A worker acquires a lock, pauses during garbage collection, loses the lease and later resumes. Another worker has already taken ownership, yet the stale worker can still write unless the protected resource rejects it.
Why the simple answer is not enough
Mutual exclusion inside one process is straightforward. Across machines, network delay means a client cannot know whether it still owns a lock.
Exactly one scheduler is often unnecessary. At-least-once scheduling plus idempotent jobs can be simpler and safer than leader election.
System flow
request lease→Coordinator
issue fencing token→Worker
perform bounded work→Resource
reject stale token
Architecture decisions
Prefer ownership partitioning
Assign each aggregate or shard to one worker and rebalance explicitly, reducing lock frequency.
Use monotonic fencing
Every lease acquisition receives a larger token; storage compares it atomically with the last accepted token.
Bound the critical section
Set a deadline below lease duration and stop accepting new sub-work when renewal becomes uncertain.
Going deeper
Time is not ownership
TTL limits how long others wait, but clock time alone cannot revoke code already running. The storage-side token closes that gap.
Locks must not hide product conflicts
Two users editing the same document may require optimistic versioning and conflict UX, not an infrastructure lock.
Implementation path
- First remove unnecessary shared ownership through partitioning or idempotency.
- Use a time-bounded lease with an increasing fencing token.
- Make the protected resource reject tokens older than the last accepted value.
Technical example: Fenced update
lease = coordinator.acquire(resourceId, ttl = 10s)
result = compute(deadline = lease.expiresAt - 2s)
UPDATE resource
SET value = :result, fence = :token
WHERE id = :id AND fence < :tokenA successful coordinator lock without a fenced write is insufficient because a paused former owner can resume.
Failure modes to design for
- Stop-the-world pause exceeds the lease.
- Coordinator quorum is unavailable and clients disagree about ownership.
- A long external call finishes after the lease has moved.
What to observe
| Signal | Why it matters |
|---|---|
| Lease acquisition and renewal failure | Shows contention and coordinator health. |
| Rejected stale fencing tokens | Proves stale owners occur and are being contained. |
| Critical-section duration | Finds work that no longer fits the lease assumptions. |
How to validate the design
- Pause a lock holder beyond TTL and then resume it.
- Partition the holder from the coordinator but not from storage.
- Run concurrent contenders and prove only increasing tokens commit.
Safe rollout plan
Instrument current critical sections, add idempotency and optimistic versions first, then introduce fencing for the few operations that require exclusive ownership. Keep a kill switch that falls back to safe serial processing.
Production checklist
- Clock assumptions are documented and minimised.
- Lease renewal failure stops new work before expiry.
- Long operations have checkpoints rather than one oversized critical section.
Takeaway
Locks coordinate attempts; fencing at the resource protects correctness.