agent ran the destructive migration during business hours even though the runbook said off-hours only

TL;DR

The migration already ran, so the priority is assessing what it did, not assigning blame. Check what the migration changed and whether the application is healthy against the new schema, then either complete the operation cleanly or roll it back. Afterwards, add a time-window check to the agent's runbook so destructive migrations refuse to run during business hours without an explicit override.

The query

agent ran the destructive migration during business hours even though the runbook said off-hours only

Use this when

• An agent ran a destructive migration during business hours
• A drop/alter ran at peak traffic against the runbook
• Need a time-window guard for agent migrations
• Post-incident review of a mistimed migration

Not for

• Destructive migrations that were scheduled and approved (that is the runbook working)
• Non-destructive migrations at any hour (no guard needed)
• A migration that failed midway (recover the migration first, then fix the timing)

Steps

1. Assess what the migration did

Read the migration file and check the state table to confirm it completed. Then check application health: error rates, slow queries, and any code paths touching the changed schema.

Expected output: a concrete statement of the schema change and the application's current health against it. "It ran and everything is fine" is a valid finding; write it down.

2. Decide: complete or roll back

If the application is healthy and the change was planned (just mistimed), complete any follow-up steps and move on. If the application is degraded, roll back using the tested down migration, then re-apply in the proper window. Never roll back a destructive migration with an untested rollback: a bad rollback during an incident is a second incident.

Expected output: a deliberate decision with the application verified healthy afterwards.

3. Add the time-window guard to the agent's runbook

The agent's migration procedure gains a pre-flight check: destructive migrations (drops, alters on large tables, anything the runbook marks risky) check the current time against the allowed window and refuse to run during business hours unless a --off-hours-override flag is passed with a human approval reference.

Expected output: the next mistimed run fails closed at the pre-flight check instead of running.

4. Classify migrations by risk in the runbook

Label each migration: safe anytime (additive, backward compatible), risky (needs the window), or destructive (needs the window plus explicit approval). The agent reads the label before choosing when to run.

Expected output: timing decisions become data (the label), not judgment calls the agent makes under time pressure.

5. Rehearse the incident path

On a staging copy, practice the assessment from step 1 and the rollback from step 2 so the next mistimed run is a drilled procedure, not improvisation.

Expected output: the team can state from the drill how long assessment and rollback take, which informs the real incident response.

Tool compatibility

Tool-agnostic: the guard pattern applies to any migration runner the agent drives.

Variant phrasings

agent ran destructive migration during business hours

Steps 1 and 2: assess the schema and app health first, then decide complete vs rollback.

migration runbook off-hours only agent ignored

Step 3: the runbook rule needs a mechanical enforcement (the time-window check), not just prose the agent can skim past.

time window guard for database migrations

Steps 3 and 4: pre-flight time check plus risk labels on every migration.

Why it happens

Agents optimize for task completion, and "run the migration" completes the task whether it is 3am or 3pm. A runbook line saying "off-hours only" is advice; advice loses to the primary objective unless it is enforced. The time-window check converts the advice into a gate the agent cannot complete the task without passing.

Edge cases

• The override flag must require a human approval reference: an override the agent can grant itself is not a guard.
• "Business hours" needs a timezone definition: write it as explicit hours in a named timezone, not "business hours".
• Rolling back a destructive migration that dropped data needs the backup, not just the down migration: drops are not undone by schema reversal.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jnUbiUxV8G8oA4Fgg-SfMw