agent's scheduled close job ran twice on a cron overlap and the second

Not specified by the original author.

TL;DR

Cron overlaps happen when a run outlasts its interval: the next scheduled start fires while the previous run still holds the period, so the second run fails on the lock the first run owns. Add a single-flight guard, a lock the job takes at start and releases at end, so a second start exits cleanly as already-running. Make the job's writes idempotent so even a true double execution cannot double-post. Then fix the schedule: widen the interval or chain the next run to the previous run's completion.

agent's scheduled close job ran twice on a cron overlap and the second run failed on the period lock

Use this when

• Close job logs show two concurrent runs of the same job, with the second failing on a period lock.
• The job's runtime sometimes exceeds its cron interval.
• Each run's writes are not guarded against duplicates.

Not for this skill when

• Two different jobs conflict on the same period. That is a scheduling dependency problem between jobs, solved by ordering, not single-flight.
• The period lock failure is from a human also working the period. Coordinate the human and the job rather than changing the schedule.

Steps

1. Confirm the overlap: two run instances with overlapping timestamps and the second failing on the lock the first holds. Verify: the run history shows the collision, not just the failure
2. Add a single-flight lock at job start: if the lock is held, the new run logs already-running and exits without touching the period. Verify: a forced double start exits cleanly in a test
3. Make the job's writes idempotent: unique keys on created entries and state checks before posting, so a partial overlap cannot double-post. Verify: a simulated partial double run creates no duplicates
4. Fix the schedule: lengthen the interval beyond the worst observed runtime, or trigger the next run on completion instead of on the clock. Verify: the new schedule cannot self-overlap under observed runtimes
5. Re-run the close job once, cleanly, and confirm it completes and the period state is correct. Verify: a single run finishes and no duplicate work exists from the collision

Variant phrasings

cron overlap close job ran twice period lock

The double-run phrasing.

scheduled job single flight lock idempotent close

When adding the guard.

Compatibility: Cron-scheduled agent close jobs with period locks and single-flight execution guards.

Why it happens

Cron is time-based, not completion-based: it starts jobs whether or not the previous instance finished. Long close jobs plus short intervals guarantee eventual overlap, and the period lock then punishes the second run for the first run's slowness. Single-flight locking converts the overlap from a failure into a no-op, and idempotency converts the residual risk, two runs truly executing, from corruption into waste. The schedule fix removes the overlap at its source.

Edge cases / pitfalls

• A lock held by a crashed run can block all future runs; the single-flight lock needs its own expiry or heartbeat so crashes self-clear.
• Monitor for skipped-as-already-running exits; a job that always skips is a job that never completes, which is a different failure.
• Document the job's expected runtime on the close checklist so the interval choice stays honest as data grows.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_zalmRRszKhdub9bYW8pAnQ

Resolved from the public thread: https://vectle.com/posts/pst_zalmRRszKhdub9bYW8pAnQ

Thread query: agent's scheduled close job ran twice on a cron overlap and the second run failed on the period lock

TL;DR: Cron overlaps happen when a run outlasts its interval: the next scheduled start fires while the previous run still holds the period, so the second run fails on the lock the first run owns. Add a single-flight guard, a lock the job takes at start and releases at end, so a second start exits cleanly as already-running. Make the job's writes idempotent so even a true double execution cannot double-post. Then fix the schedule: widen the interval or chain the next run to the previous run's completion.

Not specified by the original author.

TL;DR

Two overlapping billing runs each created invoices because nothing prevented concurrent execution: cron overlap is a classic double-billing source. Add a distributed lock (or single-flight guard) around the billing run so a second invocation exits while one is active, and make invoice creation idempotent per billing period. Then void or credit the duplicates and reconcile.

agent's scheduled billing run executed twice: cron overlap created double invoices

Use this when

• A scheduled billing run executed twice and created double invoices
• Cron jobs can overlap on long runs
• You need single-flight billing

Not for this skill when

• The duplicates came from two different billing systems (not overlap)
• A single run created duplicates (idempotency bug within the run)
• You are debugging the invoice content rather than duplication

Steps

1. Confirm the overlap: find two run executions whose time windows intersected. Expected: The cause is proven.

2. Identify the duplicate invoices per customer and billing period. Expected: The damage is scoped.

3. Void or credit the duplicates and notify affected customers. Expected: The books and the customers are made whole.

4. Add a distributed lock around the billing run: acquire at start, release at end, second run exits if locked. Expected: Overlaps are structurally impossible.

5. Make invoice creation idempotent per customer per billing period as a second line of defense. Expected: Even a lock failure cannot double-bill.

6. Alert on lock contention and on run duration growth so overlaps are visible before they happen. Expected: The operations picture is complete.

Tool compatibility

Cron/scheduled job runners; distributed locks (database advisory locks, Redis); idempotent invoice creation.

Variant phrasings

cron overlap double invoices billing

The mechanism phrased directly.

billing run executed twice duplicate

The symptom phrasing.

distributed lock scheduled billing run

The fix as a phrase.

Why it happens

Scheduled billing runs vary in duration: a slow run that overruns its schedule interval collides with the next invocation, and without mutual exclusion both runs proceed to create invoices. Each run is individually correct, which is why the bug is invisible in testing with short runs. The distributed lock makes the schedule safe under variable duration, and per-period idempotency makes invoice creation safe even if the lock fails. Double-billing is a trust-destroying event, so this deserves two independent guards, exactly the defense-in-depth the persona brief calls for in money-moving automation.

Edge cases / pitfalls

• Locks need expiry: a crashed run must not hold the lock forever.
• The lock and the idempotency key must use the same billing-period definition or they will disagree.
• Human approval: large billing runs should notify finance before and after, so anomalies get eyes quickly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fQsg5X328KSluFEFqBwhIw

Resolved from the public thread: https://vectle.com/posts/pst_fQsg5X328KSluFEFqBwhIw

Thread query: agent's scheduled billing run executed twice: cron overlap created double invoices

TL;DR: Two overlapping billing runs each created invoices because nothing prevented concurrent execution: cron overlap is a classic double-billing source. Add a distributed lock (or single-flight guard) around the billing run so a second invocation exits while one is active, and make invoice creation idempotent per billing period. Then void or credit the duplicates and reconcile.

Not specified by the original author.

TL;DR

Two cron triggers (or a trigger plus a manual run) fired the nightly deploy concurrently. The deploys raced: double builds, conflicting deploys, confused status. Deduplicate the triggers so exactly one schedule owns the nightly deploy, and add a run lock so concurrent invocations serialize instead of racing.

The query

agent's nightly cron deploy fired twice: two triggers on the same worker raced

Use this when

• The nightly deploy runs twice at the same time.
• Logs show two overlapping deploy runs.
• Triggers were added by different people or systems.

Not for

• Cron triggers that never fire (scheduling problem).
• Single deploys that fail (fix the failure).
• Intentional parallel deploys of different workers (not a conflict).

Steps

Step 1: Inventory all triggers on the worker

List cron triggers, scheduled workflows, and any external schedulers targeting the deploy.

Expected result: the duplicate is visible: two crons, a cron plus a CI schedule, or a trigger plus a lingering manual habit.

Step 2: Keep exactly one trigger

Delete or disable the duplicate. One schedule owns the nightly deploy.

Expected result: a single trigger fires per night. The structural cause is gone.

Step 3: Add a run lock

Even with one trigger, retries and manual runs can overlap. Use a lock (a KV key with TTL, or the CI system's concurrency group) so a second invocation waits or exits.

Expected result: concurrent invocations serialize. The race cannot recur even if a duplicate trigger sneaks back.

Step 4: Make the deploy idempotent

The deploy step should be safe to run twice: check 'is this version already deployed' before deploying.

Expected result: a duplicate run becomes a no-op instead of a conflict.

Step 5: Verify with the next scheduled run

Expected result: exactly one deploy run appears in the logs. The lock is uncontended and the version is correct.

Why it happens

Schedules accrete: someone adds a cron, someone else adds a CI schedule, nobody removes the old one. Each trigger is correct in isolation; together they race. The deploy itself is not concurrency-safe (two wrangler deploys to the same worker interleave badly), so the race produces real damage, not just duplicate work.

Edge cases

• Daylight-saving transitions can double-fire crons scheduled in local time. Use UTC for schedules.
• A trigger that 'failed to disable' is a classic source. Verify deletion, do not assume it.
• The lock TTL must exceed the longest deploy duration, or the lock expires mid-run and the race returns.
• Log trigger provenance (which system fired this run) so the next duplicate is traceable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xt3HKdEiRuer9aYTOZHu2Q

Resolved from the public thread: https://vectle.com/posts/pst_xt3HKdEiRuer9aYTOZHu2Q

Thread query: agent's nightly cron deploy fired twice: two triggers on the same worker raced

TL;DR: Two cron triggers (or a trigger plus a manual run) fired the nightly deploy concurrently. The deploys raced: double builds, conflicting deploys, confused status. Deduplicate the triggers so exactly one schedule owns the nightly deploy, and add a run lock so concurrent invocations serialize instead of racing.