agent's scheduler failed by firing the restore test twice concurrently

TL;DR

Double-firing comes from scheduler overlap: a run that outlives its interval gets launched again, or two agent instances both believe they own the schedule. Add a lock keyed by test and window that a second launcher cannot take, make test startup check for an already-running instance of itself, and alert on lock contention instead of silently running twice.

agent's scheduler failed by firing the restore test twice concurrently

Use this when

• Two identical restore tests run at the same time and interfere.
• Overlap happens on slow tests that exceed the schedule interval.
• A second agent instance was added for reliability and doubled the work.

Not for this skill when

• The backup product own schedule runs a job twice. Different scheduler.
• An operator manually starts a test while the scheduled one runs. Coordination, handled by the same lock.

Steps

1. Confirm from run logs that two executions overlapped, with their instance identifiers. Verify: double execution is evidenced, not suspected
2. Implement a lock per test and scheduled window using the orchestrator store or a lock file with atomic acquisition. Verify: a second launch attempt is refused in testing
3. Make the test check for a live prior instance before starting heavy work as a second line of defence. Verify: a manually started duplicate exits cleanly
4. Decide and document the missed-run policy when a lock blocks a scheduled run: skip, queue or alert. Verify: the policy is implemented and visible in reports
5. Run the schedule through a deliberately slow test. Verify: no overlap occurs and the next run behaves per policy

Variant phrasings

scheduler double fire concurrent run lock agent

The locking phrasing.

same test runs twice overlapping schedule

The overlap variant.

cron overlap long job second instance lock

The interval wording.

Compatibility: Agent schedulers using cron, systemd timers, Celery beat, Kubernetes CronJobs and custom orchestrators for restore tests.

Why it happens

Schedulers promise to start work, not to ensure the previous work finished, and restore tests are exactly the kind of long task that outlives tidy intervals. Redundancy makes it worse when a standby agent cannot see the primary is mid-run. Mutual exclusion keyed to the work, not the machine, is the standard answer.

Edge cases / pitfalls

• Kubernetes CronJobs have a concurrency policy setting for exactly this. Use it where available.
• A lock without a heartbeat becomes a permanent block after a crash. Expire locks with owner checks.
• Double runs of restore tests can corrupt each other by mounting the same restore point twice. Treat overlap as a correctness bug, not a nuisance.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__mPu7TgHe770Q2xn9UU8iA