scraping scheduler fired overlapping runs after agent missed the lock release

TL;DR

Guard every scheduled run with a distributed lock that has a TTL and a unique owner id, released in a teardown block so exceptions cannot orphan it. Long runs renew the lease with a heartbeat; a stalled run loses the lock instead of blocking everything. Overlapping runs become a skip log line, not a pileup.

scraping scheduler fired overlapping runs after agent missed the lock release

Use this when

• scheduled jobs overlap after a crash or an unhandled exception
• two runs process the same work and duplicate results
• the scheduler has no record of what is currently running
• you need at-most-one execution across restarts

Not for this skill when

• the job logic itself is broken or slow
• you want parallel runs by design (use sharded locks instead)
• the queue is backlogged for throughput reasons

Steps

1. Take a distributed lock with a TTL longer than the longest expected run, and stamp it with a unique owner id. Expected: only one run holds the lock at a time.
2. Release the lock in a teardown block (finally or equivalent) so exceptions cannot orphan it. Expected: the next run starts on time after a crash.
3. Renew the lease from a heartbeat thread for long runs, and stop the run if renewal fails. Expected: a stalled run loses the lock instead of blocking everything.
4. Before starting work, check for a live lock and skip with a logged already-running message instead of piling on. Expected: overlapping runs become a log line, not a pileup.
5. Keep the TTL short enough that a truly dead agent only blocks one cycle. Expected: the worst case is one skipped run.

Variant phrasings

scheduler fired overlapping runs

Short version: TTL lock with owner id, released in teardown, skip if held.

agent missed the lock release after a crash

Short version: release in a teardown block so exceptions cannot orphan the lock.

two runs processed the same work

Short version: check for a live lock before starting and skip with a log line.

Compatibility: works with Redis, database advisory locks, or any lock store with TTL; the pattern is the same in Python, Node, or Go, no library pinned.

Why it happens

The lock release lived on the happy path, after the work completed. When the run crashed or threw, the release never executed and the lock stayed held or, worse, was never taken consistently, so the next tick started a second run on the same work. The scheduler trusted a lock that nobody reliably released.

Edge cases / pitfalls

• A TTL shorter than the run causes the lock to expire mid-work and a second run to start; size the TTL with headroom and renew it.
• Owner ids must be unique per run; reusing a static id lets a new run release another run's lock.
• Heartbeat renewal needs its own error handling; a dead heartbeat thread silently turns the lock into a time bomb.
• Skip-if-running hides a perpetually stuck job; alert when skips happen several cycles in a row.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_q2fRsM4NAitaJgVX5zeX-g