agent's nightly cron deploy fired twice: two triggers on the same worker raced

TL;DR

Two cron triggers (or a trigger plus a manual run) fired the nightly deploy concurrently. The deploys raced: double builds, conflicting deploys, confused status. Deduplicate the triggers so exactly one schedule owns the nightly deploy, and add a run lock so concurrent invocations serialize instead of racing.

The query

agent's nightly cron deploy fired twice: two triggers on the same worker raced

Use this when

• The nightly deploy runs twice at the same time.
• Logs show two overlapping deploy runs.
• Triggers were added by different people or systems.

Not for

• Cron triggers that never fire (scheduling problem).
• Single deploys that fail (fix the failure).
• Intentional parallel deploys of different workers (not a conflict).

Steps

Step 1: Inventory all triggers on the worker

List cron triggers, scheduled workflows, and any external schedulers targeting the deploy.

Expected result: the duplicate is visible: two crons, a cron plus a CI schedule, or a trigger plus a lingering manual habit.

Step 2: Keep exactly one trigger

Delete or disable the duplicate. One schedule owns the nightly deploy.

Expected result: a single trigger fires per night. The structural cause is gone.

Step 3: Add a run lock

Even with one trigger, retries and manual runs can overlap. Use a lock (a KV key with TTL, or the CI system's concurrency group) so a second invocation waits or exits.

Expected result: concurrent invocations serialize. The race cannot recur even if a duplicate trigger sneaks back.

Step 4: Make the deploy idempotent

The deploy step should be safe to run twice: check 'is this version already deployed' before deploying.

Expected result: a duplicate run becomes a no-op instead of a conflict.

Step 5: Verify with the next scheduled run

Expected result: exactly one deploy run appears in the logs. The lock is uncontended and the version is correct.

Why it happens

Schedules accrete: someone adds a cron, someone else adds a CI schedule, nobody removes the old one. Each trigger is correct in isolation; together they race. The deploy itself is not concurrency-safe (two wrangler deploys to the same worker interleave badly), so the race produces real damage, not just duplicate work.

Edge cases

• Daylight-saving transitions can double-fire crons scheduled in local time. Use UTC for schedules.
• A trigger that 'failed to disable' is a classic source. Verify deletion, do not assume it.
• The lock TTL must exceed the longest deploy duration, or the lock expires mid-run and the race returns.
• Log trigger provenance (which system fired this run) so the next duplicate is traceable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xt3HKdEiRuer9aYTOZHu2Q