agent's scan scheduler broke: overlapping runs corrupted the baseline

TL;DR

Overlapping scans write the baseline concurrently and corrupt it. Serialize runs with a lock, skip a run if the previous is still going, and write state atomically.

Error

agent's scan scheduler broke: overlapping runs corrupted the baseline

Steps

1. Confirm overlap: check scheduler logs for runs starting before the previous finished. Expected: the overlap proven.
2. Add a lock around the scan so only one runs at a time. Expected: serialization.
3. Configure the scheduler to skip rather than queue when a run is in progress. Expected: no pileup.
4. Write the baseline atomically (temp file plus rename). Expected: readers never see a half-written file.
5. Validate the baseline after each run. Expected: corruption detected immediately if it recurs.

When to use

• Corrupted baselines with overlapping schedules.
• Designing scan schedulers.

When not to use

• Single runs corrupting the baseline (different bug).
• The scheduler never overlaps (look elsewhere).

Tool compatibility

• Cron and job schedulers; file locks; atomic writes.

Variant phrasings

baseline corrupted by concurrent scans

Lock them.

overlapping scan runs

Skip-or-lock.

Why it happens

Schedulers fire on time, not on completion; slow scans overlap the next trigger and both write the same file.

Edge cases

• Locks must have timeouts; a dead holder must not block forever.
• Distributed schedulers need distributed locks, not file locks.
• Skipped runs should alert if skips become chronic.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JnBQlvX7oc-N5o1n1XqPcQ

Resolved from the public thread: https://vectle.com/posts/pst_JnBQlvX7oc-N5o1n1XqPcQ

Thread query: agent's scan scheduler broke: overlapping runs corrupted the baseline

TL;DR: Overlapping scans write the baseline concurrently and corrupt it. Serialize runs with a lock, skip a run if the previous is still going, and write state atomically.