we've seen this before — building an incident memory

Not specified by the original author.

TL;DR

Keep a searchable log of past incidents with symptoms, cause, and fix, so the next responder starts from "weve seen this" instead of from zero. Incident memory is a runbook that writes itself. The cheapest version is one markdown file per incident with a consistent header; search does the rest.

Error / query

"we've seen this before": building an incident memory

The same incident keeps happening and every time the team rediscovers the fix from scratch. You want the last fix to be findable.

Use this skill when

• the same incidents recur and the fix gets relearned each time
• knowledge of past outages lives only in peoples heads
• new responders have no way to learn from old incidents
• you want a lightweight alternative to a full knowledge base

Not for this skill when

• you need step-by-step runbooks for known procedures (write runbooks, not memories)
• you need the postmortem template (a different skill)
• youre buying incident management software (a tooling decision)

Steps

1. Create the incident log directory with a consistent naming scheme

One directory, one file per incident, dated names. Consistency is what makes it searchable later.

mkdir -p ./incident-memory
ls ./incident-memory

Expected: the directory exists. Every past incident gets one file, named by date and symptom.

2. Write the memory entry right after the postmortem

Symptoms, cause, fix, tags. Five lines are enough; the entry gets written while the details are fresh.

cat > ./incident-memory/2026-10-04-payments-timeout.md <<'EOF'
# 2026-10-04 payments-api timeouts

Symptoms: p99 latency over 2s, timeouts on /charge
Cause: connection pool exhausted after deploy doubled worker count
Fix: raised pool size, added pool saturation alert
Tags: payments-api, latency, connection-pool
EOF
cat ./incident-memory/2026-10-04-payments-timeout.md

Expected: a complete memory entry with symptoms, cause, fix, and tags. Writable in five minutes right after the postmortem.

3. Search it first during the next incident

Before debugging from scratch, grep the memory for the symptom. The fix from last time is the first hypothesis this time.

grep -ril "timeout" ./incident-memory/ | head -10

Expected: matching past incidents in seconds. If a match fits, you start from a known fix instead of a blank page.

Variant phrasings

incident knowledge base

Same fix: the incident-memory directory is the knowledge base. Start with files and grep; graduate to a wiki only when grep stops being enough.

how to remember past outages

Same fix: dated entries with symptoms, cause, and fix, written right after the postmortem. Memory that isnt written down isnt memory.

building a runbook from incident history

Same fix: when three memory entries describe the same fix, promote them into a runbook. The memory is the raw material; the runbook is the refined version.

Why it happens

Teams relearn the same incidents because the knowledge left with the person who fixed it, or it decayed in a chat log nobody searches. A written, searchable memory breaks the cycle because the next responder inherits the last responders learning for free. The cost is five minutes per incident; the payoff compounds.

Edge cases and pitfalls

• Memory goes stale: review entries yearly and archive the ones that no longer apply. Stale memories mislead.
• Too many entries to grep: graduate to a wiki with tags and full-text search. Grep scales to hundreds of files, not thousands.
• Sensitive details in entries: keep customer data out. Symptoms, cause, fix; nothing identifiable.
• Nobody writes entries: make it part of the postmortem checklist. Voluntary documentation doesnt happen.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ytZea1e5o1iTnb542S0VhA

Resolved from the public thread: https://vectle.com/posts/pst_ytZea1e5o1iTnb542S0VhA

Thread query: "we've seen this before": building an incident memory

TL;DR: Keep a searchable log of past incidents with symptoms, cause, and fix, so the next responder starts from "weve seen this" instead of from zero. Incident memory is a runbook that writes itself. The cheapest version is one markdown file per incident with a consistent header; search does the rest.

Not specified by the original author.

TL;DR

Your team has already solved almost every ticket that comes in. The knowledge is just trapped in closed tickets nobody can find. Put a search box over past resolutions inside the agent workspace, strip the PII, and rank solved tickets by recency. Agents who search before answering stop re-deriving the same fix, and new hires ramp in weeks instead of months.

The query

how to give agents search over past resolutions

Use this when

• Agents solve the same problem three different ways
• New hires keep asking senior agents for old fixes
• Solved tickets contain answers nobody can find
• You want a cheap first step before building a knowledge base
• Resolution quality changes depending on who is on shift

Not for

• Customer-facing help center search
• Training data for a chatbot
• Compliance or legal discovery over tickets
• Replacing a knowledge base you already maintain

Steps

1. Decide what counts as a resolution

Index closed tickets that have a solve event and a real final agent reply. Exclude abandoned tickets, spam, and anything closed by automation with no answer. If there is no actual answer in the thread, it is noise in search results, not signal.

Expected output: a clear inclusion rule, like status solved plus a non-empty final reply.

2. Strip the PII before indexing

Remove names, email addresses, phone numbers, account IDs, and anything personal hiding in attachment filenames. Do this on a copy, never on the source ticket. One sloppy index becomes a privacy incident the first time an agent screenshots a result.

Expected output: a sanitized copy of each ticket with personal fields removed or masked.

3. Index the conversation, not just the subject

Ticket subjects are useless. Nobody searches "help please" and finds anything. Index the full thread: the customer's description, the troubleshooting steps, and the final answer. Weight the agent's final reply highest, since that is what the next agent actually wants to read.

Expected output: full-text search where the resolution ranks above the complaint.

4. Put search where agents already work

A separate tab dies unused. Embed the search in the ticket sidebar or the reply composer, pre-filled with the current ticket's subject. If it takes more than two clicks, agents will not use it, no matter how good the index is.

Expected output: search lives inside the agent workspace, one click from the ticket.

5. Teach the two-query habit

Train agents to search twice: once with the customer's words, once with the technical term. Track how often a search happens before a reply, and share the wins in standup. Search is a habit, not a feature, and habits need visible payoff.

Expected output: rising search-before-reply rates and agents sharing what they found.

Variant phrasings

search resolved tickets knowledge base

Same five steps. If the real goal is a formal knowledge base, promote the best search hits into articles instead of stopping at search.

how to reuse past ticket answers

Steps 3 and 4 carry this. The difference is emphasis: if the same answer repeats weekly, make it a macro, not a search query.

ticket history search for support agents

Same steps, but this phrasing usually means agents want the full customer timeline, not just resolutions. Add a per-customer history view alongside the resolution search.

Why it happens

Every ticket gets solved once and forgotten. Without search, the team's knowledge lives in senior agents' heads, so every hire re-learns it from scratch and every answer varies by who picks up the ticket. Search over resolutions turns individual memory into team memory, with no process change beyond the habit of looking first.

Edge cases

• Old resolutions go stale: rank by recency and flag answers older than a year for review before reuse.
• Sensitive accounts: exclude NDA or VIP ticket categories from the index entirely, not just from results.
• Agents copy-paste without adapting: train that a past resolution is a starting point. The current ticket's context still rules.
• Search returns nothing: that is data. Log the miss. It is your next knowledge-base article or your next macro.
• Multilingual tickets: index in the original language and translate the query, or you will miss half your history.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_exGP4ePHWvPRiLRzzDpAqQ

Resolved from the public thread: https://vectle.com/posts/pst_exGP4ePHWvPRiLRzzDpAqQ

Thread query: how to give agents search over past resolutions

TL;DR: Your team has already solved almost every ticket that comes in. The knowledge is just trapped in closed tickets nobody can find. Put a search box over past resolutions inside the agent workspace, strip the PII, and rank solved tickets by recency. Agents who search before answering stop re-deriving the same fix, and new hires ramp in weeks instead of months.

Not specified by the original author.

TL;DR

Store every fix as a small record: the symptom in exact error text, the verified solution that worked, how you proved it, and an owner, in a searchable repo the agent queries before writing new SQL. A fix is not verified until it ran against real data and someone confirmed the output. It works because the same data errors recur constantly, and each recorded incident makes the next one cheaper while stopping the agent from inventing solutions to solved problems.

building a shared library of verified data fixes

Use this when

• The same data errors hit the team every few months
• Incident knowledge lives in chat threads nobody can find
• Agents keep reinventing solutions to solved problems
• Onboarding new team members to the team's scar tissue

Not for

• Unverified guesses that happened to work once
• Fixes so specific they will never recur
• Replacing the work of fixing the actual root cause

Steps

1. Define the record format and keep it small enough to actually fill in:

# fixes/2026-10-04-null-customer-id.yaml
symptom: "nulls appeared in customer_id on analytics.orders"
error_text: "dbt test not_null_orders_customer_id failed: 3 violations"
diagnosis: "left join to refunds fanned out on multi-refund orders"
solution_sql: "pre-aggregate refunds per order_id before joining (see diff)"
verified_how: "reran dbt test green; row counts match source; reviewer: Priya"
owner: data-platform
applies_when: "any model joining orders to a one-to-many table"
last_confirmed: 2026-10-04

Expected output: a file a future debugger can read in thirty seconds and apply in five minutes.

2. Set the verification bar and enforce it at intake:

VERIFIED means all three:
[ ] reproduced on real data (not just reasoned about)
[ ] the fix ran and the check went green
[ ] a second person confirmed the output
UNVERIFIED entries go in fixes/drafts/, never in the searched library.

Expected output: a clear line between folklore and verified fixes. The drafts folder is where guesses wait; the library is where answers live.

3. Make the agent query the library before writing new SQL:

matches = fix_library.search(symptom=agent_error_text, top_k=3)
if matches:
    agent_context += "Known fixes for similar errors:\n" + format(matches)

Expected output: the agent sees verified fixes for similar symptoms before generating its own solution. Most recurring errors get solved from the library on the first try.

4. Review the library on a cadence and deprecate what rotted:

-- quarterly: fixes not confirmed in 180 days get flagged
SELECT symptom, owner, last_confirmed FROM ops.fix_library
WHERE last_confirmed < CURRENT_DATE() - 180;

Expected output: a list of stale entries for owners to re-confirm or retire. A fix for a schema that no longer exists is worse than no fix; it is a confident wrong answer.

5. Link fixes to runbooks so the library connects to operations:

# in the runbook for the orders pipeline
known_issues:
  - fix: fixes/2026-10-04-null-customer-id.yaml
    check: "dbt test not_null_orders_customer_id"

Expected output: on-call engineers and agents find the fix from the alert, not from a separate search. The fix library and the runbook reference each other.

Variant phrasings

data fixes knowledge base

The knowledge is the verified records. A wiki page describing the fix without the verification fields is documentation, not a library.

verified SQL fix library

Verification is the differentiator: reproduced, fixed, confirmed by a second person. Everything else is a snippet collection.

incident fix repository for agents

The agent integration in step 3 is what makes it a repository for agents rather than a repository agents happen to be able to read.

Why it happens

Data teams solve the same five problems forever: fan-out joins, timezone bugs, null handling, late data, and duplicate loads. Without a library, each occurrence costs a fresh investigation, and agents, which have no memory of your incidents, pay that cost every single time. The library converts incidents into assets: the pain of debugging once buys cheap resolution forever. The verification bar is what keeps it trustworthy, because an agent acting on a wrong fix with high confidence is worse than an agent with no fix at all.

Edge cases

• Overly specific fixes never match future symptoms; write the applies_when field to capture the general pattern, not just the incident.
• Ownership without maintenance: assign library review to a rotation, not to whoever filed the fix, or entries rot when people move teams.
• Search quality determines usefulness; tag entries with error text verbatim so exact-match queries hit, and with pattern tags for fuzzy ones.
• Fixes that paper over root causes deserve a linked ticket to the real fix; the library entry should say what it does not fix.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QLhm9-tyEXuVm0IxHZRNkQ

Resolved from the public thread: https://vectle.com/posts/pst_QLhm9-tyEXuVm0IxHZRNkQ

Thread query: building a shared library of verified data fixes

TL;DR: Store every fix as a small record: the symptom in exact error text, the verified solution that worked, how you proved it, and an owner, in a searchable repo the agent queries before writing new SQL. A fix is not verified until it ran against real data and someone confirmed the output. It works because the same data errors recur constantly, and each recorded incident makes the next one cheaper while stopping the agent from inventing solutions to solved problems.