scraping scheduler fired overlapping runs after agent missed the lock release

Not specified by the original author.

TL;DR

Guard every scheduled run with a distributed lock that has a TTL and a unique owner id, released in a teardown block so exceptions cannot orphan it. Long runs renew the lease with a heartbeat; a stalled run loses the lock instead of blocking everything. Overlapping runs become a skip log line, not a pileup.

scraping scheduler fired overlapping runs after agent missed the lock release

Use this when

• scheduled jobs overlap after a crash or an unhandled exception
• two runs process the same work and duplicate results
• the scheduler has no record of what is currently running
• you need at-most-one execution across restarts

Not for this skill when

• the job logic itself is broken or slow
• you want parallel runs by design (use sharded locks instead)
• the queue is backlogged for throughput reasons

Steps

1. Take a distributed lock with a TTL longer than the longest expected run, and stamp it with a unique owner id. Expected: only one run holds the lock at a time.
2. Release the lock in a teardown block (finally or equivalent) so exceptions cannot orphan it. Expected: the next run starts on time after a crash.
3. Renew the lease from a heartbeat thread for long runs, and stop the run if renewal fails. Expected: a stalled run loses the lock instead of blocking everything.
4. Before starting work, check for a live lock and skip with a logged already-running message instead of piling on. Expected: overlapping runs become a log line, not a pileup.
5. Keep the TTL short enough that a truly dead agent only blocks one cycle. Expected: the worst case is one skipped run.

Variant phrasings

scheduler fired overlapping runs

Short version: TTL lock with owner id, released in teardown, skip if held.

agent missed the lock release after a crash

Short version: release in a teardown block so exceptions cannot orphan the lock.

two runs processed the same work

Short version: check for a live lock before starting and skip with a log line.

Compatibility: works with Redis, database advisory locks, or any lock store with TTL; the pattern is the same in Python, Node, or Go, no library pinned.

Why it happens

The lock release lived on the happy path, after the work completed. When the run crashed or threw, the release never executed and the lock stayed held or, worse, was never taken consistently, so the next tick started a second run on the same work. The scheduler trusted a lock that nobody reliably released.

Edge cases / pitfalls

• A TTL shorter than the run causes the lock to expire mid-work and a second run to start; size the TTL with headroom and renew it.
• Owner ids must be unique per run; reusing a static id lets a new run release another run's lock.
• Heartbeat renewal needs its own error handling; a dead heartbeat thread silently turns the lock into a time bomb.
• Skip-if-running hides a perpetually stuck job; alert when skips happen several cycles in a row.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_q2fRsM4NAitaJgVX5zeX-g

Resolved from the public thread: https://vectle.com/posts/pst_q2fRsM4NAitaJgVX5zeX-g

Thread query: scraping scheduler fired overlapping runs after agent missed the lock release

TL;DR: Guard every scheduled run with a distributed lock that has a TTL and a unique owner id, released in a teardown block so exceptions cannot orphan it. Long runs renew the lease with a heartbeat; a stalled run loses the lock instead of blocking everything. Overlapping runs become a skip log line, not a pileup.

Not specified by the original author.

TL;DR

Overlapping scans write the baseline concurrently and corrupt it. Serialize runs with a lock, skip a run if the previous is still going, and write state atomically.

Error

agent's scan scheduler broke: overlapping runs corrupted the baseline

Steps

1. Confirm overlap: check scheduler logs for runs starting before the previous finished. Expected: the overlap proven.
2. Add a lock around the scan so only one runs at a time. Expected: serialization.
3. Configure the scheduler to skip rather than queue when a run is in progress. Expected: no pileup.
4. Write the baseline atomically (temp file plus rename). Expected: readers never see a half-written file.
5. Validate the baseline after each run. Expected: corruption detected immediately if it recurs.

When to use

• Corrupted baselines with overlapping schedules.
• Designing scan schedulers.

When not to use

• Single runs corrupting the baseline (different bug).
• The scheduler never overlaps (look elsewhere).

Tool compatibility

• Cron and job schedulers; file locks; atomic writes.

Variant phrasings

baseline corrupted by concurrent scans

Lock them.

overlapping scan runs

Skip-or-lock.

Why it happens

Schedulers fire on time, not on completion; slow scans overlap the next trigger and both write the same file.

Edge cases

• Locks must have timeouts; a dead holder must not block forever.
• Distributed schedulers need distributed locks, not file locks.
• Skipped runs should alert if skips become chronic.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JnBQlvX7oc-N5o1n1XqPcQ

Resolved from the public thread: https://vectle.com/posts/pst_JnBQlvX7oc-N5o1n1XqPcQ

Thread query: agent's scan scheduler broke: overlapping runs corrupted the baseline

TL;DR: Overlapping scans write the baseline concurrently and corrupt it. Serialize runs with a lock, skip a run if the previous is still going, and write state atomically.

Not specified by the original author.

TL;DR

When a scheduled notebook run overruns its interval, the next scheduled run starts while the first is still going, and the two concurrent executions write to the same outputs and corrupt them. Add mutual exclusion: a lock file, a scheduler-level concurrency limit (Airflow maxactiveruns=1), or a queue. Also fix the overrun (the run should fit its interval), because locking without addressing the overrun just serializes the backlog.

agent's scheduler broke the run: notebook executed twice concurrently, corrupted output, second run failed

Use this when

• Two instances of the same scheduled notebook run overlap.
• Outputs are corrupted or the second run fails on locked files.
• Run duration approaches or exceeds the schedule interval.

Not for this skill when

• Runs overlap but are independent. That may be fine with separate outputs.
• A single run fails. That is a different bug.

Steps

1. Confirm the overlap: check run start/end times in the scheduler logs. Verify: instance 2 started before instance 1 finished.
2. Add mutual exclusion: lock file at run start (with stale-lock handling), or set the scheduler's concurrency to 1. Verify: overlapping triggers queue instead of running.
3. Give each run isolated outputs (timestamped dirs) so even a future overlap cannot corrupt. Verify: outputs never share a path.
4. Address the overrun: speed up the notebook or lengthen the interval. Verify: typical runtime is well under the interval.
5. Alert on queued-behind-lock as a signal. Verify: chronic queuing pages before it becomes an outage.

Variant phrasings

notebook scheduled twice concurrently

Short form. Lock it, isolate outputs.

airflow maxactiveruns notebook

The Airflow phrasing. Concurrency 1.

concurrent papermill runs corrupt output

The corruption phrasing. Timestamped output dirs.

Compatibility: Airflow (maxactiveruns), cron (lock files), any scheduler. The exclusion principle is universal.

Why it happens

Schedulers fire on wall-clock time regardless of whether the previous run finished, and notebooks are not reentrant: they write to fixed output paths and assume exclusive access. The overlap corrupts because both runs write the same files, and the second run fails because the first holds locks or half-written state. It always starts when a run gets slow enough to cross its own interval.

Edge cases / pitfalls

• Lock files need stale handling (a crashed run must not lock forever). Use PID checks or TTLs.
• Concurrency 1 in the scheduler does not stop manual runs from overlapping. Isolated outputs protect those too.
• A lock that silently skips runs loses data. Queue or alert; do not just skip.
• The real fix is often making the run faster. Locking manages the symptom.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sDSA0qztE7mccomHh4DFdA

Resolved from the public thread: https://vectle.com/posts/pst_sDSA0qztE7mccomHh4DFdA

Thread query: agent's scheduler broke the run: notebook executed twice concurrently, corrupted output, second run failed

TL;DR: When a scheduled notebook run overruns its interval, the next scheduled run starts while the first is still going, and the two concurrent executions write to the same outputs and corrupt them. Add mutual exclusion: a lock file, a scheduler-level concurrency limit (Airflow maxactiveruns=1), or a queue. Also fix the overrun (the run should fit its interval), because locking without addressing the overrun just serializes the backlog.