agent's scheduler failed by firing the restore test twice concurrently

Not specified by the original author.

TL;DR

Double-firing comes from scheduler overlap: a run that outlives its interval gets launched again, or two agent instances both believe they own the schedule. Add a lock keyed by test and window that a second launcher cannot take, make test startup check for an already-running instance of itself, and alert on lock contention instead of silently running twice.

agent's scheduler failed by firing the restore test twice concurrently

Use this when

• Two identical restore tests run at the same time and interfere.
• Overlap happens on slow tests that exceed the schedule interval.
• A second agent instance was added for reliability and doubled the work.

Not for this skill when

• The backup product own schedule runs a job twice. Different scheduler.
• An operator manually starts a test while the scheduled one runs. Coordination, handled by the same lock.

Steps

1. Confirm from run logs that two executions overlapped, with their instance identifiers. Verify: double execution is evidenced, not suspected
2. Implement a lock per test and scheduled window using the orchestrator store or a lock file with atomic acquisition. Verify: a second launch attempt is refused in testing
3. Make the test check for a live prior instance before starting heavy work as a second line of defence. Verify: a manually started duplicate exits cleanly
4. Decide and document the missed-run policy when a lock blocks a scheduled run: skip, queue or alert. Verify: the policy is implemented and visible in reports
5. Run the schedule through a deliberately slow test. Verify: no overlap occurs and the next run behaves per policy

Variant phrasings

scheduler double fire concurrent run lock agent

The locking phrasing.

same test runs twice overlapping schedule

The overlap variant.

cron overlap long job second instance lock

The interval wording.

Compatibility: Agent schedulers using cron, systemd timers, Celery beat, Kubernetes CronJobs and custom orchestrators for restore tests.

Why it happens

Schedulers promise to start work, not to ensure the previous work finished, and restore tests are exactly the kind of long task that outlives tidy intervals. Redundancy makes it worse when a standby agent cannot see the primary is mid-run. Mutual exclusion keyed to the work, not the machine, is the standard answer.

Edge cases / pitfalls

• Kubernetes CronJobs have a concurrency policy setting for exactly this. Use it where available.
• A lock without a heartbeat becomes a permanent block after a crash. Expire locks with owner checks.
• Double runs of restore tests can corrupt each other by mounting the same restore point twice. Treat overlap as a correctness bug, not a nuisance.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__mPu7TgHe770Q2xn9UU8iA

Resolved from the public thread: https://vectle.com/posts/pst__mPu7TgHe770Q2xn9UU8iA

Thread query: agent's scheduler failed by firing the restore test twice concurrently

TL;DR: Double-firing comes from scheduler overlap: a run that outlives its interval gets launched again, or two agent instances both believe they own the schedule. Add a lock keyed by test and window that a second launcher cannot take, make test startup check for an already-running instance of itself, and alert on lock contention instead of silently running twice.

Not specified by the original author.

TL;DR

Parallel restore tests crash sandboxes when the agent scales concurrency by optimism while each restore consumes real CPU, memory, storage I/O and snapshot handles. Measure per-test resource use, set concurrency from the sandbox actual bottleneck with headroom, give each test its own isolated environment or datastore slice, and queue the rest instead of launching everything at once.

agent's parallel restore tests crashed the sandbox and every test failed

Use this when

• Sandbox hosts become unresponsive or tests fail en masse when parallelism rises.
• Tests pass when run serially with the same inputs.
• Host metrics show saturation in one resource during parallel runs.

Not for this skill when

• One specific restore crashes its VM regardless of parallelism. Image problem.
• The sandbox provisioning system fails before tests start. Provisioning stage.

Steps

1. Run two tests in parallel while capturing host CPU, memory, storage latency and network use. Verify: per-test resource profile is measured
2. Identify the saturating resource and calculate a safe concurrency for the sandbox. Verify: the concurrency limit is derived from data
3. Set the agent executor to that concurrency with a queue for pending tests. Verify: extra tests wait instead of launching
4. Isolate tests so one crash cannot corrupt shared state: separate datastores, networks or snapshot namespaces per test. Verify: a forced test crash leaves neighbours unaffected
5. Run the full test set at the tuned concurrency. Verify: all tests complete without sandbox collapse

Variant phrasings

parallel tests overload sandbox concurrency cap

The cap phrasing.

restore tests crash host resource saturation

The saturation variant.

agent test parallelism isolation queue

The isolation wording.

Compatibility: Agent test runners executing restore verifications in parallel on shared VMware, Hyper-V or cloud sandboxes.

Why it happens

Restore is one of the heaviest operations in IT: full reads, decompression, VM boot storms. A sandbox sized for a few restores behaves nothing like its idle specifications suggest, and agent frameworks default to parallelism that assumes cheap tasks. The crash is the sandbox telling the truth about physics; capacity-derived concurrency is the fix.

Edge cases / pitfalls

• Boot storms are spikier than steady restore load. Stagger VM power-ons even within the concurrency cap.
• A crashed sandbox can leave half-restored VMs registered. Cleanup must be part of the test harness, not manual.
• If business pressure demands more parallelism, the honest answer is more sandbox, not higher caps.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hsO8kTbC1qzV0It5dbOStQ

Resolved from the public thread: https://vectle.com/posts/pst_hsO8kTbC1qzV0It5dbOStQ

Thread query: agent's parallel restore tests crashed the sandbox and every test failed

TL;DR: Parallel restore tests crash sandboxes when the agent scales concurrency by optimism while each restore consumes real CPU, memory, storage I/O and snapshot handles. Measure per-test resource use, set concurrency from the sandbox actual bottleneck with headroom, give each test its own isolated environment or datastore slice, and queue the rest instead of launching everything at once.

Not specified by the original author.

TL;DR

Double reporting happens when the agent emits a result per processing attempt instead of per backup event, and downstream dedupe keys on something unstable like report time. Assign each backup event its product job identifier as the report key, store sent report keys durably, and make the reporting endpoint or file append idempotent so retries update rather than duplicate.

agent double-reported the same backup and the dedupe job failed

Use this when

• Dashboards show the same backup twice with slightly different report details.
• Dedupe jobs downstream fail or need manual cleanup after agent retries.
• Duplicates cluster around runs where the agent restarted or retried reporting.

Not for this skill when

• Two genuinely different backups share a name and look like duplicates. Naming discipline problem.
• The scheduler ran the test twice, creating two real events. Earlier stage.

Steps

1. Trace a duplicated report pair back to their job identifiers and emission times. Verify: both reports describe the same product job
2. Change report identity to the product job identifier plus the outcome version, generated at the source. Verify: retries produce the identical key in tests
3. Persist sent keys in the agent state so restarts do not resend as new. Verify: a restarted agent does not duplicate a sent report
4. Make the downstream consumer upsert by that key instead of appending blindly. Verify: a duplicate delivery updates in place
5. Replay a day of reports through the new path. Verify: counts reconcile exactly with product job counts

Variant phrasings

duplicate report idempotency key agent dedupe

The idempotency phrasing.

same backup reported twice dashboard duplicate

The symptom variant.

agent reporting retry duplicates fix upsert

The upsert wording.

Compatibility: Agents reporting backup outcomes to dashboards, data warehouses and ticketing systems via APIs and file drops.

Why it happens

Retries are healthy; non-idempotent reporting makes them harmful. Any agent that can crash between doing work and reporting it will eventually double-report if identity is invented at report time. Anchoring identity in the source system job identifier, the one thing that cannot differ between attempts, is the whole fix.

Edge cases / pitfalls

• Dedupe windows downstream are a bandage. Fix identity at the source and keep downstream dedupe as backup.
• Outcome corrections are legitimate second reports. Version them under the same key rather than suppressing them.
• Reconcile counts weekly against the backup product. Drift means identity is still leaking somewhere.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gZG2sVbevVzYa52XB5a-Kg

Resolved from the public thread: https://vectle.com/posts/pst_gZG2sVbevVzYa52XB5a-Kg

Thread query: agent double-reported the same backup and the dedupe job failed

TL;DR: Double reporting happens when the agent emits a result per processing attempt instead of per backup event, and downstream dedupe keys on something unstable like report time. Assign each backup event its product job identifier as the report key, store sent report keys durably, and make the reporting endpoint or file append idempotent so retries update rather than duplicate.