## TL;DR

An incident is over when the customer-facing impact has stopped for a sustained period (error rate back to baseline for 30 minutes is a good bar), the fix is verified by someone other than the person who applied it, and monitoring shows no recurrence. Saying "its fixed" after one green check is how incidents reopen. Write the criteria down before you need them.

## Error / query

```text
how to declare an incident over: resolution criteria
```

The graphs look better and someone typed "looks fixed" in the channel. You need a real bar for done, not optimism.

## Use this skill when

- youre closing an incident and want criteria instead of gut feel
- incidents keep reopening an hour after the all-clear
- youre writing runbooks and need a resolution checklist
- theres disagreement about whether its actually over

## Not for this skill when

- you need the postmortem template (that comes after resolution)
- the incident is still actively being debugged (work the incident first)
- you need escalation policy (see the escalation skill)

## Steps

### 1. Verify impact is gone from the customers perspective

Check the customer-facing metrics against the pre-incident baseline, not just the internal fix. Your deploy succeeding is not the same as customers recovering.

```bash
curl -s "http://YOUR-prometheus-host/api/v1/query?query=up" | head -c 200
```

Expected: targets report up, and your error-rate and latency dashboards match the pre-incident baseline. Internal "deploy succeeded" is not enough.

### 2. Get a second pair of eyes on the fix

The person who applied the fix is the worst person to verify it. Run the checklist with someone else.

```bash
cat > /tmp/resolution-checklist.txt <<'EOF'
[ ] impact stopped for 30+ min on customer-facing metrics
[ ] fix verified by someone other than the fixer
[ ] no recurrence in the last 30 min of logs and alerts
[ ] incident commander agrees it is over
EOF
cat /tmp/resolution-checklist.txt
```

Expected: all four boxes checked by two people. If any box is unchecked, the incident is not over.

### 3. Announce the all-clear with a timestamp and schedule the postmortem

Close the loop publicly so nobody is left wondering, and get the postmortem on the calendar while memories are fresh.

```bash
echo "Incident declared over at $(date -u +%Y-%m-%dT%H:%M:%SZ). Postmortem scheduled within 48h."
```

Expected: a timestamped all-clear in the incident channel and a postmortem on the calendar. Resolution sticks because the follow-up is already scheduled.

## Variant phrasings

### when to close an incident

Same fix: close it when the step 2 checklist is fully checked, not when the graphs first turn green.

### incident resolution checklist

Same fix: the four boxes in step 2 are the checklist. Print it, check it, keep it.

### how to know an outage is really fixed

Same fix: sustained baseline on customer metrics plus independent verification. One green check is hope; thirty green minutes is evidence.

## Why it happens

Incidents reopen because "looks fine now" isnt verification, and because theres social pressure to declare victory and stand down. A written bar removes the pressure: the checklist says when its over, not the loudest voice in the channel. The 30-minute watch window exists because most recurrences happen inside it.

## Edge cases and pitfalls

- Intermittent issues: extend the watch window to cover a few full cycles of the flakiness, not just 30 minutes.
- Partial mitigation: declare "mitigated", not "resolved", and keep the incident open with reduced severity.
- The fix caused a second incident: reopen the original or open a linked one; dont stack silent follow-ups.
- Customer reports after the all-clear: treat as a new incident or reopen based on linkage, but decide explicitly and say so.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gGpjXdXmQQkEPZp4MLjpIA