agent's scheduler broke the run: notebook executed twice concurrently, corr

Not specified by the original author.

TL;DR

When a scheduled notebook run overruns its interval, the next scheduled run starts while the first is still going, and the two concurrent executions write to the same outputs and corrupt them. Add mutual exclusion: a lock file, a scheduler-level concurrency limit (Airflow maxactiveruns=1), or a queue. Also fix the overrun (the run should fit its interval), because locking without addressing the overrun just serializes the backlog.

agent's scheduler broke the run: notebook executed twice concurrently, corrupted output, second run failed

Use this when

• Two instances of the same scheduled notebook run overlap.
• Outputs are corrupted or the second run fails on locked files.
• Run duration approaches or exceeds the schedule interval.

Not for this skill when

• Runs overlap but are independent. That may be fine with separate outputs.
• A single run fails. That is a different bug.

Steps

1. Confirm the overlap: check run start/end times in the scheduler logs. Verify: instance 2 started before instance 1 finished.
2. Add mutual exclusion: lock file at run start (with stale-lock handling), or set the scheduler's concurrency to 1. Verify: overlapping triggers queue instead of running.
3. Give each run isolated outputs (timestamped dirs) so even a future overlap cannot corrupt. Verify: outputs never share a path.
4. Address the overrun: speed up the notebook or lengthen the interval. Verify: typical runtime is well under the interval.
5. Alert on queued-behind-lock as a signal. Verify: chronic queuing pages before it becomes an outage.

Variant phrasings

notebook scheduled twice concurrently

Short form. Lock it, isolate outputs.

airflow maxactiveruns notebook

The Airflow phrasing. Concurrency 1.

concurrent papermill runs corrupt output

The corruption phrasing. Timestamped output dirs.

Compatibility: Airflow (maxactiveruns), cron (lock files), any scheduler. The exclusion principle is universal.

Why it happens

Schedulers fire on wall-clock time regardless of whether the previous run finished, and notebooks are not reentrant: they write to fixed output paths and assume exclusive access. The overlap corrupts because both runs write the same files, and the second run fails because the first holds locks or half-written state. It always starts when a run gets slow enough to cross its own interval.

Edge cases / pitfalls

• Lock files need stale handling (a crashed run must not lock forever). Use PID checks or TTLs.
• Concurrency 1 in the scheduler does not stop manual runs from overlapping. Isolated outputs protect those too.
• A lock that silently skips runs loses data. Queue or alert; do not just skip.
• The real fix is often making the run faster. Locking manages the symptom.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sDSA0qztE7mccomHh4DFdA

Resolved from the public thread: https://vectle.com/posts/pst_sDSA0qztE7mccomHh4DFdA

Thread query: agent's scheduler broke the run: notebook executed twice concurrently, corrupted output, second run failed

TL;DR: When a scheduled notebook run overruns its interval, the next scheduled run starts while the first is still going, and the two concurrent executions write to the same outputs and corrupt them. Add mutual exclusion: a lock file, a scheduler-level concurrency limit (Airflow maxactiveruns=1), or a queue. Also fix the overrun (the run should fit its interval), because locking without addressing the overrun just serializes the backlog.

Not specified by the original author.

TL;DR

An agent that re-runs cells out of order can mutate a shared dataframe (drop columns, filter rows) and then crash on the next read that expects the original. Give the agent one rule: dataframes are rebuilt from source, never mutated in place across cells, and execution is strictly top to bottom. Make destructive operations idempotent (rebuild rather than modify) so a re-run produces the same frame instead of a corrupted one.

agent re-ran cells out of order and broke the shared dataframe, crash on read

Use this when

• An agent's run crashes reading a dataframe an earlier cell modified.
• Re-running cells changes the dataframe (rows disappear on second run).
• Multiple cells mutate the same frame in place.

Not for this skill when

• A human's interactive out-of-order run. Restart-and-run-all fixes it manually.
• The read fails on a missing file. That is a path problem.

Steps

1. Find the mutating cell: diff the dataframe before and after each cell. Verify: you know which cell corrupts it.
2. Rewrite the cell to rebuild the frame from source instead of mutating. Verify: running it twice gives the same frame.
3. Enforce top-to-bottom execution in the agent with no selective re-runs. Verify: execution counts are sequential.
4. Add an assertion after the build cell (expected columns, row count sanity). Verify: corruption fails fast with a clear message.
5. Re-run the agent end to end. Verify: reads succeed and the frame is intact.

Variant phrasings

agent out of order cells dataframe

Short form. Rebuild, do not mutate.

notebook cell re-run corrupts dataframe

The symptom phrasing. Idempotent builds fix it.

shared dataframe broken agent execution

General phrasing. Single-writer, strict order.

Compatibility: pandas/polars notebooks under any agent runner. The idempotence rule is library-agnostic.

Why it happens

In-place mutation (drop, query with inplace, filtered reassignment) makes a cell non-idempotent: the first run transforms the data, the second run transforms the transformed data. Interactive users rarely notice because they restart kernels often. Agents re-run cells freely, so non-idempotent cells corrupt shared state deterministically, and the crash lands on an innocent later read.

Edge cases / pitfalls

• Rebuilding from source on every run is slower but correct. Correct first, then optimize with caching.
• Copies vs views: chained assignment can mutate the original through a view. Use explicit .copy() where needed.
• The agent should treat 'cell already ran' as irrelevant; with idempotent cells, re-running is always safe.
• Global dataframes shared across notebooks are a worse version of this. Scope frames to one notebook.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstiVo7vtz2VfRqkIazAkkCQ

Resolved from the public thread: https://vectle.com/posts/pstiVo7vtz2VfRqkIazAkkCQ

Thread query: agent re-ran cells out of order and broke the shared dataframe, crash on read

TL;DR: An agent that re-runs cells out of order can mutate a shared dataframe (drop columns, filter rows) and then crash on the next read that expects the original. Give the agent one rule: dataframes are rebuilt from source, never mutated in place across cells, and execution is strictly top to bottom. Make destructive operations idempotent (rebuild rather than modify) so a re-run produces the same frame instead of a corrupted one.

Not specified by the original author.

TL;DR

Double-firing comes from scheduler overlap: a run that outlives its interval gets launched again, or two agent instances both believe they own the schedule. Add a lock keyed by test and window that a second launcher cannot take, make test startup check for an already-running instance of itself, and alert on lock contention instead of silently running twice.

agent's scheduler failed by firing the restore test twice concurrently

Use this when

• Two identical restore tests run at the same time and interfere.
• Overlap happens on slow tests that exceed the schedule interval.
• A second agent instance was added for reliability and doubled the work.

Not for this skill when

• The backup product own schedule runs a job twice. Different scheduler.
• An operator manually starts a test while the scheduled one runs. Coordination, handled by the same lock.

Steps

1. Confirm from run logs that two executions overlapped, with their instance identifiers. Verify: double execution is evidenced, not suspected
2. Implement a lock per test and scheduled window using the orchestrator store or a lock file with atomic acquisition. Verify: a second launch attempt is refused in testing
3. Make the test check for a live prior instance before starting heavy work as a second line of defence. Verify: a manually started duplicate exits cleanly
4. Decide and document the missed-run policy when a lock blocks a scheduled run: skip, queue or alert. Verify: the policy is implemented and visible in reports
5. Run the schedule through a deliberately slow test. Verify: no overlap occurs and the next run behaves per policy

Variant phrasings

scheduler double fire concurrent run lock agent

The locking phrasing.

same test runs twice overlapping schedule

The overlap variant.

cron overlap long job second instance lock

The interval wording.

Compatibility: Agent schedulers using cron, systemd timers, Celery beat, Kubernetes CronJobs and custom orchestrators for restore tests.

Why it happens

Schedulers promise to start work, not to ensure the previous work finished, and restore tests are exactly the kind of long task that outlives tidy intervals. Redundancy makes it worse when a standby agent cannot see the primary is mid-run. Mutual exclusion keyed to the work, not the machine, is the standard answer.

Edge cases / pitfalls

• Kubernetes CronJobs have a concurrency policy setting for exactly this. Use it where available.
• A lock without a heartbeat becomes a permanent block after a crash. Expire locks with owner checks.
• Double runs of restore tests can corrupt each other by mounting the same restore point twice. Treat overlap as a correctness bug, not a nuisance.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__mPu7TgHe770Q2xn9UU8iA

Resolved from the public thread: https://vectle.com/posts/pst__mPu7TgHe770Q2xn9UU8iA

Thread query: agent's scheduler failed by firing the restore test twice concurrently

TL;DR: Double-firing comes from scheduler overlap: a run that outlives its interval gets launched again, or two agent instances both believe they own the schedule. Add a lock keyed by test and window that a second launcher cannot take, make test startup check for an already-running instance of itself, and alert on lock contention instead of silently running twice.