we've seen this before": building an incident memory

TL;DR

Keep a searchable log of past incidents with symptoms, cause, and fix, so the next responder starts from "weve seen this" instead of from zero. Incident memory is a runbook that writes itself. The cheapest version is one markdown file per incident with a consistent header; search does the rest.

Error / query

"we've seen this before": building an incident memory

The same incident keeps happening and every time the team rediscovers the fix from scratch. You want the last fix to be findable.

Use this skill when

• the same incidents recur and the fix gets relearned each time
• knowledge of past outages lives only in peoples heads
• new responders have no way to learn from old incidents
• you want a lightweight alternative to a full knowledge base

Not for this skill when

• you need step-by-step runbooks for known procedures (write runbooks, not memories)
• you need the postmortem template (a different skill)
• youre buying incident management software (a tooling decision)

Steps

1. Create the incident log directory with a consistent naming scheme

One directory, one file per incident, dated names. Consistency is what makes it searchable later.

mkdir -p ./incident-memory
ls ./incident-memory

Expected: the directory exists. Every past incident gets one file, named by date and symptom.

2. Write the memory entry right after the postmortem

Symptoms, cause, fix, tags. Five lines are enough; the entry gets written while the details are fresh.

cat > ./incident-memory/2026-10-04-payments-timeout.md <<'EOF'
# 2026-10-04 payments-api timeouts

Symptoms: p99 latency over 2s, timeouts on /charge
Cause: connection pool exhausted after deploy doubled worker count
Fix: raised pool size, added pool saturation alert
Tags: payments-api, latency, connection-pool
EOF
cat ./incident-memory/2026-10-04-payments-timeout.md

Expected: a complete memory entry with symptoms, cause, fix, and tags. Writable in five minutes right after the postmortem.

3. Search it first during the next incident

Before debugging from scratch, grep the memory for the symptom. The fix from last time is the first hypothesis this time.

grep -ril "timeout" ./incident-memory/ | head -10

Expected: matching past incidents in seconds. If a match fits, you start from a known fix instead of a blank page.

Variant phrasings

incident knowledge base

Same fix: the incident-memory directory is the knowledge base. Start with files and grep; graduate to a wiki only when grep stops being enough.

how to remember past outages

Same fix: dated entries with symptoms, cause, and fix, written right after the postmortem. Memory that isnt written down isnt memory.

building a runbook from incident history

Same fix: when three memory entries describe the same fix, promote them into a runbook. The memory is the raw material; the runbook is the refined version.

Why it happens

Teams relearn the same incidents because the knowledge left with the person who fixed it, or it decayed in a chat log nobody searches. A written, searchable memory breaks the cycle because the next responder inherits the last responders learning for free. The cost is five minutes per incident; the payoff compounds.

Edge cases and pitfalls

• Memory goes stale: review entries yearly and archive the ones that no longer apply. Stale memories mislead.
• Too many entries to grep: graduate to a wiki with tags and full-text search. Grep scales to hundreds of files, not thousands.
• Sensitive details in entries: keep customer data out. Symptoms, cause, fix; nothing identifiable.
• Nobody writes entries: make it part of the postmortem checklist. Voluntary documentation doesnt happen.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ytZea1e5o1iTnb542S0VhA