agent's scheduler broke the run: notebook executed twice concurrently, corrupted output, second run failed

TL;DR

When a scheduled notebook run overruns its interval, the next scheduled run starts while the first is still going, and the two concurrent executions write to the same outputs and corrupt them. Add mutual exclusion: a lock file, a scheduler-level concurrency limit (Airflow maxactiveruns=1), or a queue. Also fix the overrun (the run should fit its interval), because locking without addressing the overrun just serializes the backlog.

agent's scheduler broke the run: notebook executed twice concurrently, corrupted output, second run failed

Use this when

• Two instances of the same scheduled notebook run overlap.
• Outputs are corrupted or the second run fails on locked files.
• Run duration approaches or exceeds the schedule interval.

Not for this skill when

• Runs overlap but are independent. That may be fine with separate outputs.
• A single run fails. That is a different bug.

Steps

1. Confirm the overlap: check run start/end times in the scheduler logs. Verify: instance 2 started before instance 1 finished.
2. Add mutual exclusion: lock file at run start (with stale-lock handling), or set the scheduler's concurrency to 1. Verify: overlapping triggers queue instead of running.
3. Give each run isolated outputs (timestamped dirs) so even a future overlap cannot corrupt. Verify: outputs never share a path.
4. Address the overrun: speed up the notebook or lengthen the interval. Verify: typical runtime is well under the interval.
5. Alert on queued-behind-lock as a signal. Verify: chronic queuing pages before it becomes an outage.

Variant phrasings

notebook scheduled twice concurrently

Short form. Lock it, isolate outputs.

airflow maxactiveruns notebook

The Airflow phrasing. Concurrency 1.

concurrent papermill runs corrupt output

The corruption phrasing. Timestamped output dirs.

Compatibility: Airflow (maxactiveruns), cron (lock files), any scheduler. The exclusion principle is universal.

Why it happens

Schedulers fire on wall-clock time regardless of whether the previous run finished, and notebooks are not reentrant: they write to fixed output paths and assume exclusive access. The overlap corrupts because both runs write the same files, and the second run fails because the first holds locks or half-written state. It always starts when a run gets slow enough to cross its own interval.

Edge cases / pitfalls

• Lock files need stale handling (a crashed run must not lock forever). Use PID checks or TTLs.
• Concurrency 1 in the scheduler does not stop manual runs from overlapping. Isolated outputs protect those too.
• A lock that silently skips runs loses data. Queue or alert; do not just skip.
• The real fix is often making the run faster. Locking manages the symptom.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sDSA0qztE7mccomHh4DFdA