agent scheduled a Durable Object alarm test that fired the production handler

TL;DR

The agent tested alarms against the production Durable Object namespace, and the test alarm fired the real handler with real side effects (emails sent, charges made, data mutated). Stop the bleeding, audit what the handler did, and move all alarm testing to an isolated namespace with no production side effects.

The query

agent scheduled a Durable Object alarm test that fired the production handler

Use this when

• A test triggered real production side effects via a DO alarm.
• The handler cannot distinguish test from real invocations.
• The test targeted the production namespace.

Not for

• 'Alarm scheduled in the past' errors (scheduling problem, separate fix).
• Production alarms firing on their own schedule (expected behavior).
• Alarm handler bugs (fix the handler).

Steps

Step 1: Assess the side effects

Read the handler code and determine what the test invocation did: what was sent, mutated, or charged.

Expected result: a concrete blast-radius list. You cannot remediate what you have not inventoried.

Step 2: Contain and remediate

Undo what can be undone (cancel sent items where possible, reverse mutations, note charges). Notify affected parties per the incident process.

Expected result: the damage is bounded and communicated. The incident is managed, not just fixed.

Step 3: Move alarm tests to an isolated namespace

Tests run against a dedicated test DO namespace whose handler has all side effects stubbed or directed at test sinks.

Expected result: no test can reach production side effects, structurally. The isolation is in the namespace binding, not in the agent's care.

Step 4: Make the handler test-aware as defense in depth

Pass an explicit test flag through the alarm payload; the handler refuses to perform real side effects when it is set.

Expected result: even a misdirected test alarm is harmless. Two layers (isolation plus flag) protect production.

Step 5: Ban production-namespace testing in the runbook

Expected result: the agent's test flow targets the test namespace by construction. Production namespaces are never test targets.

Why it happens

Alarms are fire-and-forget with delayed effects: the agent scheduled the test, moved on, and the production handler fired minutes later. The delay decouples cause from effect, so the agent did not connect its test to the production incident. Namespaces look identical between test and prod; only the binding differs.

Edge cases

• The alarm payload should always include its provenance (test or production, which run scheduled it). Forensics without provenance is guesswork.
• One-shot alarms scheduled during a test can outlive the test run. Cancel test alarms explicitly at test teardown.
• If the handler already ran with production effects, check idempotency: rerunning remediation must not double-apply.
• Review other scheduled mechanisms (cron triggers, queues) for the same test-vs-production confusion.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FsyFIuD1RGpNWeQG9LNi6w