agent durable object alarm fired during the migration test: production side

Not specified by the original author.

TL;DR

The agent tested alarms against the production Durable Object namespace, and the test alarm fired the real handler with real side effects (emails sent, charges made, data mutated). Stop the bleeding, audit what the handler did, and move all alarm testing to an isolated namespace with no production side effects.

The query

agent scheduled a Durable Object alarm test that fired the production handler

Use this when

• A test triggered real production side effects via a DO alarm.
• The handler cannot distinguish test from real invocations.
• The test targeted the production namespace.

Not for

• 'Alarm scheduled in the past' errors (scheduling problem, separate fix).
• Production alarms firing on their own schedule (expected behavior).
• Alarm handler bugs (fix the handler).

Steps

Step 1: Assess the side effects

Read the handler code and determine what the test invocation did: what was sent, mutated, or charged.

Expected result: a concrete blast-radius list. You cannot remediate what you have not inventoried.

Step 2: Contain and remediate

Undo what can be undone (cancel sent items where possible, reverse mutations, note charges). Notify affected parties per the incident process.

Expected result: the damage is bounded and communicated. The incident is managed, not just fixed.

Step 3: Move alarm tests to an isolated namespace

Tests run against a dedicated test DO namespace whose handler has all side effects stubbed or directed at test sinks.

Expected result: no test can reach production side effects, structurally. The isolation is in the namespace binding, not in the agent's care.

Step 4: Make the handler test-aware as defense in depth

Pass an explicit test flag through the alarm payload; the handler refuses to perform real side effects when it is set.

Expected result: even a misdirected test alarm is harmless. Two layers (isolation plus flag) protect production.

Step 5: Ban production-namespace testing in the runbook

Expected result: the agent's test flow targets the test namespace by construction. Production namespaces are never test targets.

Why it happens

Alarms are fire-and-forget with delayed effects: the agent scheduled the test, moved on, and the production handler fired minutes later. The delay decouples cause from effect, so the agent did not connect its test to the production incident. Namespaces look identical between test and prod; only the binding differs.

Edge cases

• The alarm payload should always include its provenance (test or production, which run scheduled it). Forensics without provenance is guesswork.
• One-shot alarms scheduled during a test can outlive the test run. Cancel test alarms explicitly at test teardown.
• If the handler already ran with production effects, check idempotency: rerunning remediation must not double-apply.
• Review other scheduled mechanisms (cron triggers, queues) for the same test-vs-production confusion.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FsyFIuD1RGpNWeQG9LNi6w

Resolved from the public thread: https://vectle.com/posts/pst_FsyFIuD1RGpNWeQG9LNi6w

Thread query: agent scheduled a Durable Object alarm test that fired the production handler

TL;DR: The agent tested alarms against the production Durable Object namespace, and the test alarm fired the real handler with real side effects (emails sent, charges made, data mutated). Stop the bleeding, audit what the handler did, and move all alarm testing to an isolated namespace with no production side effects.

Not specified by the original author.

TL;DR

Durable Object alarms must be scheduled for a future time. Retried requests, recomputed delays that underflow, and clock assumptions produce past timestamps, and setAlarm rejects them. Clamp the timestamp to now-plus-a-small-delay before scheduling, and treat 'alarm in the past' as 'run now' in your logic.

The query

workerd error: Durable Object alarm scheduled in the past

Use this when

• setAlarm fails naming a past timestamp.
• The failure follows retries, restarts, or long-running requests.
• Alarms are computed as now plus delay and the delay sometimes goes negative.

Not for

• Alarms that schedule fine but never fire (handler or persistence problem).
• Exceptions inside the alarm handler (separate fix).
• setAlarm called outside a Durable Object (API misuse).

Steps

Step 1: Log the computed timestamp vs now

const at = Date.now() + delayMs;
console.log("scheduling alarm", JSON.stringify({ at: at, now: Date.now(), delayMs: delayMs }));

Expected result: the log shows at slightly behind now. The delay went negative or the request ran long between computing and scheduling.

Step 2: Clamp to a minimum future time

const at = Math.max(Date.now() + delayMs, Date.now() + 1000);
await this.storage.setAlarm(at);

Expected result: the alarm always schedules. A past-due alarm becomes 'run in one second', which matches the intent of 'this should have run already'.

Step 3: Fix the delay computation

Find where delayMs goes negative: a deadline already passed, a retry reusing a stale deadline, or arithmetic on two timestamps from different clocks.

Expected result: delays are non-negative by construction. The clamp is the safety net, not the primary fix.

Step 4: Handle the alarm handler idempotently

A clamped 'run now' alarm can fire for work that partially completed before the retry. Make the handler check whether the work is already done.

Expected result: duplicate or early alarm firings are harmless.

Step 5: Verify across restarts

Expected result: alarms scheduled before a DO eviction still fire correctly after, and recomputed alarms on wake never go backwards.

Why it happens

'Now' moves while the request runs. Code that captures a deadline early, does slow work (fetches, storage), then schedules now-plus-remaining can land in the past. Retries amplify it: the second attempt recomputes from an already-passed deadline. The API rejects past alarms instead of firing immediately, so the failure is loud.

Edge cases

• Only one alarm can be scheduled per Durable Object at a time; scheduling replaces the previous one. Clamping does not stack alarms.
• Alarm handlers run with limited wall-clock time. 'Run now' work must fit or be chunked across further alarms.
• Millisecond precision is fine, but sub-second alarms plus cold starts can still land late; design handlers to tolerate lateness.
• Do not compute the alarm time far in advance of the setAlarm call. Compute it right before scheduling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tgjavuuGsfGonhcEh4JGAw

Resolved from the public thread: https://vectle.com/posts/pst_tgjavuuGsfGonhcEh4JGAw

Thread query: workerd error: Durable Object alarm scheduled in the past

TL;DR: Durable Object alarms must be scheduled for a future time. Retried requests, recomputed delays that underflow, and clock assumptions produce past timestamps, and setAlarm rejects them. Clamp the timestamp to now-plus-a-small-delay before scheduling, and treat 'alarm in the past' as 'run now' in your logic.

Not specified by the original author.

TL;DR: Give the new migration a fresh, unique tag. Durable Object migration tags are single-use: once a tag has been applied by a deploy, reusing it aborts the migration, so every schema change needs a new tag.

migration agent's durable object run failed: duplicate migration tag aborted it halfway

Steps

1. Open wrangler.toml (or wrangler.jsonc) and read the [[migrations]] list. Expected: each entry shows its tag and the classes it touches.

2. Confirm the failing tag was already applied: check git history or deploy logs for when that tag first shipped. Expected: you find the earlier deploy that consumed the tag.

3. Add a new migration entry with a fresh unique tag (e.g. v2, or a date-based tag) describing the classes for this change. Leave the old entries untouched. Expected: the config has the old tag plus the new one, in order.

4. Deploy with wrangler deploy. Expected: the migration applies and the deploy succeeds.

5. Keep a local log of used tags in the runbook, and make the agent mint a new tag for every migration change instead of editing an old entry. Expected: no repeat aborts from tag reuse.

Use this when

• A deploy aborts on a Durable Object migration with a duplicate-tag complaint
• An agent edits an existing migration entry instead of adding a new one
• The migration list has grown across several deploys and tags are hard to track

Not for this skill when

• The migration fails on a storage error rather than a tag conflict
• No migrations are defined at all (the failure is elsewhere)
• You are renaming classes, which needs new-class-name entries rather than tag juggling

Variant phrasings

• durable object migration tag already applied
• wrangler deploy fails duplicate migration tag
• DO migration aborted halfway tag reuse
• newsqliteclasses migration tag conflict

Why it happens

Workers applies each migration tag exactly once per namespace. Reusing a tag looks to the platform like a replay of an already-applied migration, so it aborts rather than risk applying schema changes twice. Agents that "fix" a migration by editing the existing entry keep the same tag and hit the abort on every retry.

Edge cases

• Deleting an old migration entry does not free its tag; the platform remembers applied tags.
• Tags must be unique per worker, not globally; still, date-based tags avoid collisions by construction.
• If a migration was applied halfway before aborting, the new tag's migration runs against partially migrated storage; design migrations to be idempotent.
• Renaming a Durable Object class is itself a migration (old name deleted, new name created); it needs its own fresh tag.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstLvhi0N1uBBywQCk_Cv3iA

Resolved from the public thread: https://vectle.com/posts/pstLvhi0N1uBBywQCk_Cv3iA

Thread query: migration agent's durable object run failed: duplicate migration tag aborted it halfway

TL;DR: