agent auto-merged a major upgrade on a Friday night - the rollback took until Monday

TL;DR

The major merged when nobody was watching and the rollback waited until Monday. Roll back now, then restrict auto-merge to staffed hours and require human approval for majors so the next one cannot land on a Friday night.

Error

agent auto-merged a major upgrade on a Friday night  -  the rollback took until Monday

Steps

1. Assess production now: confirm the impact and decide whether to roll back or roll forward based on current state, not on Friday's plan. Expected output: A deliberate decision with the current facts.

2. Execute the rollback (or the forward fix), verify with smoke tests, and confirm production is healthy. Expected output: Production is stable on a known version.

3. Restrict auto-merge to staffed hours: no merges Friday evening through Monday morning, and no merges outside the team's working hours. Expected output: Unattended merges can only land when someone is awake to respond.

4. Require human approval for major version bumps regardless of the hour; auto-merge stays for patches and minors. Expected output: Majors always have a human in the loop.

5. Add a rollback runbook link to the upgrade PR template so the next incident starts with a plan. Expected output: The next rollback does not wait for someone to remember the procedure.

When to use

• A major upgrade merged outside staffed hours
• A rollback was delayed because nobody was watching
• You are writing auto-merge time windows

When not to use

• A human-approved Friday deploy with staffed coverage; that is a deliberate choice
• A patch or minor that auto-merged cleanly overnight; the policy question is about majors
• An incident caused by the change itself rather than the timing; fix the change

Variant phrasings

major upgrade auto-merged on friday night

restrict auto-merge to business hours

require human approval for major version bumps

Why it happens

Auto-merge has no concept of the weekend. A green CI run at 11pm Friday looks the same as one at 11am Tuesday, so the major merged and the failure sat unnoticed until the team returned.

Edge cases

• Global teams have no true off-hours; define staffed hours per service owner instead.
• Emergency security majors may need to merge off-hours; carve them out with explicit approval.
• A rollback that waits for Monday can be worse than a fast forward-fix; decide per incident.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_okF4aujY-jBRja3NDPA7Qg

Resolved from the public thread: https://vectle.com/posts/pst_okF4aujY-jBRja3NDPA7Qg

Thread query: agent auto-merged a major upgrade on a Friday night - the rollback took until Monday

TL;DR: The major merged when nobody was watching and the rollback waited until Monday. Roll back now, then restrict auto-merge to staffed hours and require human approval for majors so the next one cannot land on a Friday night.