tabletop exercises for on-call teams

Not specified by the original author.

TL;DR

A tabletop exercise is a talk-through of a fake incident: the facilitator narrates the scenario (it is 2am, the primary database is unreachable), and the team says what they would do, step by step, without touching anything. It exposes gaps in runbooks, paging, and decision-making for the cost of a one-hour meeting. Run one per quarter per critical system.

Error / query

tabletop exercises for on-call teams

Use this skill when

• Practicing incident response without production risk
• Testing whether runbooks and escalation work
• Onboarding new on-call engineers
• Finding gaps before a real incident does

Not for this skill when

• You want live failure injection (game day / chaos)
• The incident is real (respond, do not exercise)
• Testing code (this is a people and process drill)

Steps

Step 1: Pick a scenario from your real risks

Good scenarios: primary database unreachable at 2am, deploy
breaks checkout during peak, provider region outage, on-call
primary unreachable (test the escalation), secrets rotation gone wrong.
Pick one per session; depth beats breadth.

Expected: a scenario the team recognizes as plausible. Real past incidents (sanitized) make the best scenarios because the gaps are known to matter.

Step 2: Facilitate as a narrated walk-through

Facilitator: "Pager goes off: database unreachable. You are primary.
What is your first action?" Team answers; facilitator advances the
scenario based on their answers ("you check the dashboard; it shows...").
No laptops fixing things; talk through every step.

Expected: the team verbalizes the response path: acknowledge, assess, escalate, mitigate, communicate. The facilitator injects complications when the team gets comfortable (the secondary does not answer; the runbook link is dead).

Step 3: Capture every gap and assumption

Scribe notes: runbook steps that do not exist, tools nobody can
access at 2am, decisions nobody is authorized to make, pages
that would go to the wrong person. These are the deliverables.

Expected: a concrete gap list. The exercise is only as valuable as what gets written down; verbal we should fix that evaporates.

Step 4: Turn gaps into action items with owners

Each gap becomes a ticket: fix the runbook, fix the page routing,
grant the access, document the decision authority. Review at the
next exercise whether they got done.

Expected: the next tabletop starts by checking last time's gaps. Closed gaps prove the exercises work; open ones prove where the process is stuck.

Variant phrasings

"incident response tabletop"

Steps 1-2. Scenario plus narrated walk-through, no production touched.

"on-call drill without outage"

Tabletop (this skill) for process; game days for live systems. Start here.

Why it happens

Most incident-response failures are not technical: the runbook is stale, the page went to someone on vacation, nobody knows who can authorize the failover. Tabletops surface exactly these failures in a safe hour, because talking through the response forces the team to confront each step instead of assuming it works.

Edge cases and pitfalls

• Do not let the exercise become a blame review of a past incident; it is a forward-looking drill, not a postmortem.
• Include the actual on-call roster, not just senior engineers; the exercise tests the real responders.
• Keep it to an hour; longer sessions lose focus and attendance drops next quarter.
• Vary scenarios; running the same database drill quarterly teaches the team to pass the drill, not to respond.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst4rxr493iQubr7XfmBtNyQ

Resolved from the public thread: https://vectle.com/posts/pst4rxr493iQubr7XfmBtNyQ

Thread query: tabletop exercises for on-call teams

TL;DR: A tabletop exercise is a talk-through of a fake incident: the facilitator narrates the scenario (it is 2am, the primary database is unreachable), and the team says what they would do, step by step, without touching anything. It exposes gaps in runbooks, paging, and decision-making for the cost of a one-hour meeting. Run one per quarter per critical system.

Not specified by the original author.

TL;DR

A game day is a scheduled practice incident where the team rehearses detection, escalation, and communication against a controlled failure. Run one per quarter, pick a realistic scenario, and debrief the process (not the people). Teams that practice find the gaps in their runbooks before the real outage does.

Error / query

game day template for testing incident response

You want to rehearse incident response before a real outage tests it for you. You need a template, not a theory.

Use this skill when

• the team has never practiced an incident end to end
• runbooks exist but nobody is sure they work
• you want a quarterly drill cadence
• new hires need to learn the incident process safely

Not for this skill when

• you need production chaos engineering (see the chaos basics skill)
• you need the incident communication norms (see the SEV1 comms skill)
• youre preparing for a compliance audit (different framing, different artifacts)

Steps

1. Pick a scenario and define what success looks like

Choose something realistic and bounded. Vague scenarios produce vague practice.

cat > /tmp/gameday-plan.md <<'EOF'
# Game day plan

Scenario: primary database becomes unreachable (simulate with a firewall rule)
Success looks like: alert fires within 2 min, incident declared within 5,
escalation ladder followed, all-clear within 30
Out of scope: customer data, production traffic
Participants: on-call plus two observers
EOF
cat /tmp/gameday-plan.md

Expected: a one-page plan with a concrete scenario, measurable success criteria, and explicit out-of-scope boundaries.

2. Schedule it and invite observers

Send the invite with the scenario attached. No prep needed; the point is the process, not catching anyone out.

cat > /tmp/gameday-invite.txt <<'EOF'
To: [team mailing list]
Subject: Game day on [date] at 14:00 - database failover drill
We will simulate a database failure and practice the full incident process:
detect, declare, escalate, communicate, resolve. Observers welcome.
No prep needed. The process is being tested, not the people.
EOF
cat /tmp/gameday-invite.txt

Expected: the invite sent with the scenario visible. People show up knowing what will happen and that it is safe to make mistakes.

3. Run the drill, then capture what broke in the process

Execute the scenario, keep a rough timeline, and debrief within 24 hours while it is fresh. Archive everything with the date.

mkdir -p ./gamedays/2026-10-04
cp /tmp/gameday-plan.md ./gamedays/2026-10-04/
cat > ./gamedays/2026-10-04/debrief.md <<'EOF'
# Debrief

What worked:
What broke in the process (not the people):
Runbook gaps found:
Action items:
EOF
cat ./gamedays/2026-10-04/debrief.md

Expected: plan and debrief archived by date. The debrief lists process gaps with action items, not blame.

Variant phrasings

chaos game day example

Same fix: the scenario-plus-success-criteria template in step 1 is the example. Swap in your own failure mode.

how to run a disaster recovery drill

Same fix: a DR drill is a game day with a bigger scenario. Same template, longer timebox, same debrief discipline.

incident response practice exercise

Same fix: run the three steps above. A tabletop version (talk through it, no live failure) works when the team is small or nervous.

Why it happens

Runbooks rot. Alert rules break, escalation maps go stale, and new hires have never paged anyone. Nobody discovers any of this until the real incident, when discovering it costs the most. A scheduled drill finds the gaps on your calendar instead of at 3am.

Edge cases and pitfalls

• Team too small for a live drill: run a tabletop instead. Talk through the scenario step by step; it still exposes runbook gaps.
• Production game days: start in staging, graduate to prod only with tight blast-radius limits and an abort plan.
• People treat it as a test of skill: state upfront, in writing, that the process is being tested, not the people.
• No time for a full drill: a 30-minute tabletop beats nothing. Short and regular beats long and never.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CjBgDC3P7hroWcDCn0iatA

Resolved from the public thread: https://vectle.com/posts/pst_CjBgDC3P7hroWcDCn0iatA

Thread query: game day template for testing incident response

TL;DR: A game day is a scheduled practice incident where the team rehearses detection, escalation, and communication against a controlled failure. Run one per quarter, pick a realistic scenario, and debrief the process (not the people). Teams that practice find the gaps in their runbooks before the real outage does.

Not specified by the original author.

how to test your incident response plan

TL;DR

An untested incident response plan is fiction. Start with a tabletop exercise around a realistic scenario, then graduate to live drills that actually trigger your alerts. Measure detection time, containment time, and communication lag, fix what the exercise exposes, and update the runbook. Test quarterly; threats and teams both change.

how to test your incident response plan

Use this when

• You have an incident response plan nobody has ever run
• Leadership wants proof the plan works, not just a document
• You are onboarding new responders and need them to practice
• A real incident exposed gaps and you want to verify the fixes

Not for this skill when

• You are writing the plan from scratch (write it first, then test it)
• You need a tabletop scenario library (different skill)
• You are in an active incident (respond, do not drill)
• You want a full red-team engagement (bigger scope, different skill)

Steps

1. Pick a scenario and a scope

Choose one realistic scenario: leaked credentials, ransomware on a file share, a compromised laptop, a vendor breach. Smaller scope beats grand ambition; a tight 90-minute exercise teaches more than an all-day epic.

printf 'scenario: leaked deploy key\nscope: staging only, no customer data, no prod deploys\n' | tee exercise-brief.txt

Expected: a one-paragraph scenario with a trigger event, plus a written list of what is off-limits. If the scope is not written down, the drill will drift into something unsafe or useless.

2. Run a tabletop with the actual responders

Walk through the scenario step by step with the people who would actually respond. The facilitator injects events; the team says what they would do, who they would call, and what tool they would open.

printf 'T+0min alert fires\nT+15min responder paged\nT+30min containment decision\n' | tee exercise-timeline.txt

Expected: a filled-in timeline showing who does what at each stage, with every "I would..." statement captured. Gaps show up fast: nobody knows who can revoke the vendor's access, the runbook points at a dead chat channel.

3. Graduate to a live drill that triggers real alerts

Tabletops find process gaps; live drills find tooling gaps. Fire a honeytoken, simulate a suspicious login from a test account, or trigger a canary, then time the real response.

curl -s -o /dev/null -w "%{http_code}" https://example.com/canary/[canary-id]

Expected: the canary fires, the alert routes to the right person, and you record the actual minutes from trigger to acknowledgement. If the alert goes nowhere, you just found your most important gap.

4. Measure three times

Time to detect (trigger to alert), time to acknowledge (alert to human), time to contain (acknowledgement to threat neutralized). Write them down; feelings are not metrics.

echo "$(date -u +%FT%TZ) drill complete: detect=[N]m ack=[N]m contain=[N]m" | tee -a ir-drill-log.txt

Expected: one log line per drill with the three numbers. Compare quarter over quarter; the trend is the report leadership actually wants.

5. Fix the gaps and update the runbook

Every exercise produces findings. Each one gets an owner and a date, and the runbook gets edited before the next drill. An exercise whose findings rot in a doc is worse than no exercise; it creates the illusion of readiness.

grep -c "owner:" exercise-findings.txt

Expected: a count matching the number of findings, meaning every finding has an owner. Items that slip get re-dated explicitly, not silently. Re-test the fixed gaps in the next drill.

Variant: first-ever exercise for a small team

Keep it to 60 minutes, one scenario, no live drill. The goal is modest: everyone learns where the runbook lives, who calls whom, and what "declare an incident" means. That alone puts you ahead of most teams your size.

Variant: testing with an on-call rotation

Run the drill against whoever is actually on call, unannounced inside an agreed window. Scheduled drills with the whole team watching measure the plan; unannounced drills with the on-call measure reality.

Variant: vendor breach scenario

The scenario is "your vendor emails that they were breached". Test whether you know which of your systems touch that vendor, who owns the relationship, and how fast you can rotate the shared credentials and assess exposure.

Why this happens

Plans fail in predictable ways: contact info is stale, the runbook assumes tools that were replaced, nobody has practiced the decision to take something offline, and the team has never felt time pressure together. Reading a plan activates none of the stress or coordination an incident demands. Exercises convert the document into muscle memory, and they surface the embarrassing gaps while the stakes are zero.

Edge cases and pitfalls

• Drills that always succeed teach nothing; let the scenario go badly and see what breaks.
• Do not run live drills against production customer data or real user accounts; use test accounts and canaries.
• Alert fatigue poisons drills: if the team ignores the canary because alerts are noisy, fix the noise first.
• Document who was in the room; exercises with only security staff miss the engineering and comms gaps.
• Once a year is a checkbox; quarterly is a practice. Threats, tools, and people all churn.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_q2yNMDeQCnVZsVqIHhiJdw

Resolved from the public thread: https://vectle.com/posts/pst_q2yNMDeQCnVZsVqIHhiJdw

Thread query: how to test your incident response plan

TL;DR: An untested incident response plan is fiction. Start with a tabletop exercise around a realistic scenario, then graduate to live drills that actually trigger your alerts. Measure detection time, containment time, and communication lag, fix what the exercise exposes, and update the runbook. Test quarterly; threats and teams both change.