## TL;DR

A game day is a scheduled practice incident where the team rehearses detection, escalation, and communication against a controlled failure. Run one per quarter, pick a realistic scenario, and debrief the process (not the people). Teams that practice find the gaps in their runbooks before the real outage does.

## Error / query

```text
game day template for testing incident response
```

You want to rehearse incident response before a real outage tests it for you. You need a template, not a theory.

## Use this skill when

- the team has never practiced an incident end to end
- runbooks exist but nobody is sure they work
- you want a quarterly drill cadence
- new hires need to learn the incident process safely

## Not for this skill when

- you need production chaos engineering (see the chaos basics skill)
- you need the incident communication norms (see the SEV1 comms skill)
- youre preparing for a compliance audit (different framing, different artifacts)

## Steps

### 1. Pick a scenario and define what success looks like

Choose something realistic and bounded. Vague scenarios produce vague practice.

```bash
cat > /tmp/gameday-plan.md <<'EOF'
# Game day plan

Scenario: primary database becomes unreachable (simulate with a firewall rule)
Success looks like: alert fires within 2 min, incident declared within 5,
escalation ladder followed, all-clear within 30
Out of scope: customer data, production traffic
Participants: on-call plus two observers
EOF
cat /tmp/gameday-plan.md
```

Expected: a one-page plan with a concrete scenario, measurable success criteria, and explicit out-of-scope boundaries.

### 2. Schedule it and invite observers

Send the invite with the scenario attached. No prep needed; the point is the process, not catching anyone out.

```bash
cat > /tmp/gameday-invite.txt <<'EOF'
To: [team mailing list]
Subject: Game day on [date] at 14:00 - database failover drill
We will simulate a database failure and practice the full incident process:
detect, declare, escalate, communicate, resolve. Observers welcome.
No prep needed. The process is being tested, not the people.
EOF
cat /tmp/gameday-invite.txt
```

Expected: the invite sent with the scenario visible. People show up knowing what will happen and that it is safe to make mistakes.

### 3. Run the drill, then capture what broke in the process

Execute the scenario, keep a rough timeline, and debrief within 24 hours while it is fresh. Archive everything with the date.

```bash
mkdir -p ./gamedays/2026-10-04
cp /tmp/gameday-plan.md ./gamedays/2026-10-04/
cat > ./gamedays/2026-10-04/debrief.md <<'EOF'
# Debrief

What worked:
What broke in the process (not the people):
Runbook gaps found:
Action items:
EOF
cat ./gamedays/2026-10-04/debrief.md
```

Expected: plan and debrief archived by date. The debrief lists process gaps with action items, not blame.

## Variant phrasings

### chaos game day example

Same fix: the scenario-plus-success-criteria template in step 1 is the example. Swap in your own failure mode.

### how to run a disaster recovery drill

Same fix: a DR drill is a game day with a bigger scenario. Same template, longer timebox, same debrief discipline.

### incident response practice exercise

Same fix: run the three steps above. A tabletop version (talk through it, no live failure) works when the team is small or nervous.

## Why it happens

Runbooks rot. Alert rules break, escalation maps go stale, and new hires have never paged anyone. Nobody discovers any of this until the real incident, when discovering it costs the most. A scheduled drill finds the gaps on your calendar instead of at 3am.

## Edge cases and pitfalls

- Team too small for a live drill: run a tabletop instead. Talk through the scenario step by step; it still exposes runbook gaps.
- Production game days: start in staging, graduate to prod only with tight blast-radius limits and an abort plan.
- People treat it as a test of skill: state upfront, in writing, that the process is being tested, not the people.
- No time for a full drill: a 30-minute tabletop beats nothing. Short and regular beats long and never.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CjBgDC3P7hroWcDCn0iatA