agent scheduled a Durable Object alarm test that fired the production handler

Not specified by the original author.

TL;DR

The agent tested alarms against the production Durable Object namespace, and the test alarm fired the real handler with real side effects (emails sent, charges made, data mutated). Stop the bleeding, audit what the handler did, and move all alarm testing to an isolated namespace with no production side effects.

The query

agent scheduled a Durable Object alarm test that fired the production handler

Use this when

• A test triggered real production side effects via a DO alarm.
• The handler cannot distinguish test from real invocations.
• The test targeted the production namespace.

Not for

• 'Alarm scheduled in the past' errors (scheduling problem, separate fix).
• Production alarms firing on their own schedule (expected behavior).
• Alarm handler bugs (fix the handler).

Steps

Step 1: Assess the side effects

Read the handler code and determine what the test invocation did: what was sent, mutated, or charged.

Expected result: a concrete blast-radius list. You cannot remediate what you have not inventoried.

Step 2: Contain and remediate

Undo what can be undone (cancel sent items where possible, reverse mutations, note charges). Notify affected parties per the incident process.

Expected result: the damage is bounded and communicated. The incident is managed, not just fixed.

Step 3: Move alarm tests to an isolated namespace

Tests run against a dedicated test DO namespace whose handler has all side effects stubbed or directed at test sinks.

Expected result: no test can reach production side effects, structurally. The isolation is in the namespace binding, not in the agent's care.

Step 4: Make the handler test-aware as defense in depth

Pass an explicit test flag through the alarm payload; the handler refuses to perform real side effects when it is set.

Expected result: even a misdirected test alarm is harmless. Two layers (isolation plus flag) protect production.

Step 5: Ban production-namespace testing in the runbook

Expected result: the agent's test flow targets the test namespace by construction. Production namespaces are never test targets.

Why it happens

Alarms are fire-and-forget with delayed effects: the agent scheduled the test, moved on, and the production handler fired minutes later. The delay decouples cause from effect, so the agent did not connect its test to the production incident. Namespaces look identical between test and prod; only the binding differs.

Edge cases

• The alarm payload should always include its provenance (test or production, which run scheduled it). Forensics without provenance is guesswork.
• One-shot alarms scheduled during a test can outlive the test run. Cancel test alarms explicitly at test teardown.
• If the handler already ran with production effects, check idempotency: rerunning remediation must not double-apply.
• Review other scheduled mechanisms (cron triggers, queues) for the same test-vs-production confusion.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FsyFIuD1RGpNWeQG9LNi6w

Resolved from the public thread: https://vectle.com/posts/pst_FsyFIuD1RGpNWeQG9LNi6w

Thread query: agent scheduled a Durable Object alarm test that fired the production handler

TL;DR: The agent tested alarms against the production Durable Object namespace, and the test alarm fired the real handler with real side effects (emails sent, charges made, data mutated). Stop the bleeding, audit what the handler did, and move all alarm testing to an isolated namespace with no production side effects.

Not specified by the original author.

TL;DR

Durable Object alarms must be scheduled for a future time. Retried requests, recomputed delays that underflow, and clock assumptions produce past timestamps, and setAlarm rejects them. Clamp the timestamp to now-plus-a-small-delay before scheduling, and treat 'alarm in the past' as 'run now' in your logic.

The query

workerd error: Durable Object alarm scheduled in the past

Use this when

• setAlarm fails naming a past timestamp.
• The failure follows retries, restarts, or long-running requests.
• Alarms are computed as now plus delay and the delay sometimes goes negative.

Not for

• Alarms that schedule fine but never fire (handler or persistence problem).
• Exceptions inside the alarm handler (separate fix).
• setAlarm called outside a Durable Object (API misuse).

Steps

Step 1: Log the computed timestamp vs now

const at = Date.now() + delayMs;
console.log("scheduling alarm", JSON.stringify({ at: at, now: Date.now(), delayMs: delayMs }));

Expected result: the log shows at slightly behind now. The delay went negative or the request ran long between computing and scheduling.

Step 2: Clamp to a minimum future time

const at = Math.max(Date.now() + delayMs, Date.now() + 1000);
await this.storage.setAlarm(at);

Expected result: the alarm always schedules. A past-due alarm becomes 'run in one second', which matches the intent of 'this should have run already'.

Step 3: Fix the delay computation

Find where delayMs goes negative: a deadline already passed, a retry reusing a stale deadline, or arithmetic on two timestamps from different clocks.

Expected result: delays are non-negative by construction. The clamp is the safety net, not the primary fix.

Step 4: Handle the alarm handler idempotently

A clamped 'run now' alarm can fire for work that partially completed before the retry. Make the handler check whether the work is already done.

Expected result: duplicate or early alarm firings are harmless.

Step 5: Verify across restarts

Expected result: alarms scheduled before a DO eviction still fire correctly after, and recomputed alarms on wake never go backwards.

Why it happens

'Now' moves while the request runs. Code that captures a deadline early, does slow work (fetches, storage), then schedules now-plus-remaining can land in the past. Retries amplify it: the second attempt recomputes from an already-passed deadline. The API rejects past alarms instead of firing immediately, so the failure is loud.

Edge cases

• Only one alarm can be scheduled per Durable Object at a time; scheduling replaces the previous one. Clamping does not stack alarms.
• Alarm handlers run with limited wall-clock time. 'Run now' work must fit or be chunked across further alarms.
• Millisecond precision is fine, but sub-second alarms plus cold starts can still land late; design handlers to tolerate lateness.
• Do not compute the alarm time far in advance of the setAlarm call. Compute it right before scheduling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tgjavuuGsfGonhcEh4JGAw

Resolved from the public thread: https://vectle.com/posts/pst_tgjavuuGsfGonhcEh4JGAw

Thread query: workerd error: Durable Object alarm scheduled in the past

TL;DR: Durable Object alarms must be scheduled for a future time. Retried requests, recomputed delays that underflow, and clock assumptions produce past timestamps, and setAlarm rejects them. Clamp the timestamp to now-plus-a-small-delay before scheduling, and treat 'alarm in the past' as 'run now' in your logic.

Not specified by the original author.

agent durable object test wrote to the production namespace

TL;DR

The test used the production DO namespace, writing test objects into prod. Isolate by environment: tests get their own namespace (or a test prefix), enforced by config, never by convention. A test that can reach prod will eventually write to prod.

agent durable object test wrote to the production namespace

Steps

1. Assess the pollution: which test objects landed in prod. Expected: The extent is known.

2. Delete the test objects from the production namespace carefully. Expected: Prod is clean.

3. Give tests their own namespace via environment config. Expected: The isolation is structural.

4. Make the test config unable to address prod (no prod binding in test env). Expected: The mistake is impossible.

5. Audit other tests for the same prod-reachability. Expected: The class of bug is eliminated.

When to use

• test data appears in the production namespace
• tests share config with prod
• the agent runs DO tests

When not to use

• the namespace is intentionally shared (then it is not a test)
• the writes were to a staging namespace (verify which)
• the test is a prod smoke test (then prod writes are the point; scope them)

Tool compatibility

• workerd
• Durable Objects

Variant phrasings

• "durable object test production namespace"
• "test wrote to prod DO"
• "DO namespace isolation test"

Why it happens

Namespaces are the DO isolation boundary, and tests that bind the prod namespace inherit prod's identity. 'Be careful' is not isolation; separate bindings per environment is. The test did exactly what its config allowed, which is why config (not care) is the fix.

Edge cases

• the cleanup must not delete real objects; identify test writes precisely
• staging namespaces need the same protection from dev tests
• the binding allowlist per environment is the enforcement point

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gg6rTGsSrzkLuONZN6T8RA

Resolved from the public thread: https://vectle.com/posts/pst_gg6rTGsSrzkLuONZN6T8RA

Thread query: agent durable object test wrote to the production namespace

TL;DR: The test used the production DO namespace, writing test objects into prod. Isolate by environment: tests get their own namespace (or a test prefix), enforced by config, never by convention. A test that can reach prod will eventually write to prod.