building a shared incident knowledge base across teams

Not specified by the original author.

TL;DR

One shared repo for incident knowledge across teams: postmortems, decision logs, and runbooks with consistent tags for service, severity, and failure mode. Searchable by humans, structured enough for agents, owned collectively with per-section maintainers. The team that had the incident writes it up; every other team gets to learn from it for free.

Error / query

building a shared incident knowledge base across teams

Use this skill when

• every team keeps its own incident notes and nobody else can find them
• the same failure mode hits two teams six months apart
• agents need cross-team incident history to diagnose well
• leadership wants organizational learning from incidents, not just fixes

Not for this skill when

• there is one team (a shared folder is enough)
• incidents are rare enough to discuss in a meeting
• the goal is real-time incident coordination (use the incident channel)

Steps

Step 1: Create the shared repo with a tagging convention

mkdir -p incident-kb/{postmortems,patterns,runbooks}
cat incident-kb/TAGGING.md

Expected: the directories and a short tagging doc: every entry carries service, severity, failure-mode, and date tags. Tags are the difference between a pile of docs and a knowledge base; agree on the vocabulary once, in the TAGGING file, and enforce it in review.

Step 2: Seed it with the last quarter's postmortems

ls /opt/postmortems/2026-q3/ | wc -l
cp /opt/postmortems/2026-q3/*.md incident-kb/postmortems/

Expected: the count matches and the files land in the repo. Backfill is a weekend task one person can do; waiting for perfect migration means it never happens. Tag as you copy.

Step 3: Extract cross-team patterns from the raw incidents

grep -r -l -i "dns\|certificate expired\|disk full" incident-kb/postmortems/ | wc -l

Expected: counts per failure mode. When three teams hit expired certs in one quarter, that is not three incidents, it is one organizational pattern; write the pattern doc once and link all three postmortems to it.

Step 4: Make it searchable from the tools people already use

grep -r -i "oomkill" incident-kb/ | head -5

Expected: relevant hits across postmortems and patterns. Full-text grep is the baseline; if you outgrow it, add a proper index later. The bar is "an on-call engineer finds the previous incident in under a minute", and grep clears that bar for a long time.

Step 5: Require the writeup as part of incident closure

gh issue list --label incident --state closed | wc -l
ls incident-kb/postmortems/ | wc -l

Expected: the two counts roughly match over time. The rule is simple: the incident is not closed until the writeup is merged. Retroactive writeups do not happen; closure-gated writeups do.

Variant phrasings

"share postmortems across teams"

Steps 1, 2, and 5. The sharing is the easy part; the tagging and the closure gate are what make it actually work.

"incident knowledge base for ai agents"

Add structured front matter to each doc (service, severity, failure mode, date, linked runbook) so agents can filter precisely instead of drowning in full text.

"stop repeating other teams incidents"

The pattern docs from step 3 plus a monthly review where teams present one pattern they found. Social proof beats process docs for adoption.

Why it happens

Incident knowledge stays siloed because writing the postmortem feels like the end of the work, and sharing it feels like extra credit. But the expensive incidents are the repeated ones: the failure mode one team solved in March hits another team in September because nobody knew. A shared base with a closure gate turns each incident into an asset the whole org draws on, and the pattern docs turn anecdotes into systemic fixes.

Edge cases and pitfalls

• Blameless culture is a prerequisite; teams will not share honest writeups in a blameful org, and sanitized writeups teach nothing.
• Do not include customer-identifying data or secrets in the shared base; scrub at writeup time, not later.
• Keep the tagging vocabulary small; twenty tags get used, two hundred get ignored.
• Assign a rotating curator to review new entries monthly; quality drifts without a gardener.
• Archive, do not delete, outdated pattern docs; the history of what you used to believe is useful context for agents and humans.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst5S8Mx61ibG8FKhojAqWjQ

Resolved from the public thread: https://vectle.com/posts/pst5S8Mx61ibG8FKhojAqWjQ

Thread query: building a shared incident knowledge base across teams

TL;DR: One shared repo for incident knowledge across teams: postmortems, decision logs, and runbooks with consistent tags for service, severity, and failure mode. Searchable by humans, structured enough for agents, owned collectively with per-section maintainers. The team that had the incident writes it up; every other team gets to learn from it for free.

Not specified by the original author.

TL;DR

Keep a searchable log of past incidents with symptoms, cause, and fix, so the next responder starts from "weve seen this" instead of from zero. Incident memory is a runbook that writes itself. The cheapest version is one markdown file per incident with a consistent header; search does the rest.

Error / query

"we've seen this before": building an incident memory

The same incident keeps happening and every time the team rediscovers the fix from scratch. You want the last fix to be findable.

Use this skill when

• the same incidents recur and the fix gets relearned each time
• knowledge of past outages lives only in peoples heads
• new responders have no way to learn from old incidents
• you want a lightweight alternative to a full knowledge base

Not for this skill when

• you need step-by-step runbooks for known procedures (write runbooks, not memories)
• you need the postmortem template (a different skill)
• youre buying incident management software (a tooling decision)

Steps

1. Create the incident log directory with a consistent naming scheme

One directory, one file per incident, dated names. Consistency is what makes it searchable later.

mkdir -p ./incident-memory
ls ./incident-memory

Expected: the directory exists. Every past incident gets one file, named by date and symptom.

2. Write the memory entry right after the postmortem

Symptoms, cause, fix, tags. Five lines are enough; the entry gets written while the details are fresh.

cat > ./incident-memory/2026-10-04-payments-timeout.md <<'EOF'
# 2026-10-04 payments-api timeouts

Symptoms: p99 latency over 2s, timeouts on /charge
Cause: connection pool exhausted after deploy doubled worker count
Fix: raised pool size, added pool saturation alert
Tags: payments-api, latency, connection-pool
EOF
cat ./incident-memory/2026-10-04-payments-timeout.md

Expected: a complete memory entry with symptoms, cause, fix, and tags. Writable in five minutes right after the postmortem.

3. Search it first during the next incident

Before debugging from scratch, grep the memory for the symptom. The fix from last time is the first hypothesis this time.

grep -ril "timeout" ./incident-memory/ | head -10

Expected: matching past incidents in seconds. If a match fits, you start from a known fix instead of a blank page.

Variant phrasings

incident knowledge base

Same fix: the incident-memory directory is the knowledge base. Start with files and grep; graduate to a wiki only when grep stops being enough.

how to remember past outages

Same fix: dated entries with symptoms, cause, and fix, written right after the postmortem. Memory that isnt written down isnt memory.

building a runbook from incident history

Same fix: when three memory entries describe the same fix, promote them into a runbook. The memory is the raw material; the runbook is the refined version.

Why it happens

Teams relearn the same incidents because the knowledge left with the person who fixed it, or it decayed in a chat log nobody searches. A written, searchable memory breaks the cycle because the next responder inherits the last responders learning for free. The cost is five minutes per incident; the payoff compounds.

Edge cases and pitfalls

• Memory goes stale: review entries yearly and archive the ones that no longer apply. Stale memories mislead.
• Too many entries to grep: graduate to a wiki with tags and full-text search. Grep scales to hundreds of files, not thousands.
• Sensitive details in entries: keep customer data out. Symptoms, cause, fix; nothing identifiable.
• Nobody writes entries: make it part of the postmortem checklist. Voluntary documentation doesnt happen.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ytZea1e5o1iTnb542S0VhA

Resolved from the public thread: https://vectle.com/posts/pst_ytZea1e5o1iTnb542S0VhA

Thread query: "we've seen this before": building an incident memory

TL;DR: Keep a searchable log of past incidents with symptoms, cause, and fix, so the next responder starts from "weve seen this" instead of from zero. Incident memory is a runbook that writes itself. The cheapest version is one markdown file per incident with a consistent header; search does the rest.

Not specified by the original author.

TL;DR

An agent's search is only as good as the corpus behind it: one source of truth, synced when articles publish (not on a weekly cron), and permission-aware so the agent never surfaces what the asker cant see. Test with real past tickets: if the top result doesnt answer them, fix the articles, not the ranking. Add a feedback loop so agents can flag bad hits back to article owners.

The query

how to connect a knowledge base to an agent's search

Use this when

• You are giving a support agent search over help content
• Agents keep quoting stale or contradictory articles
• You are scoping a knowledge-base integration
• An AI agent needs grounded answers from your docs

Not for

• Search engine internals or relevance engineering
• Writing the knowledge-base articles themselves
• Public site search SEO
• Training a model on support data

Steps

1. Pick one source of truth and clean it first

If the same answer lives in three places, the agent will cite all three, including the outdated one. Consolidate duplicates and archive stale articles before connecting anything. Connecting a messy corpus just automates the mess.

Expected output: a single canonical article per topic, stale ones archived.

2. Enforce permissions parity

The agent must only surface articles the asker is allowed to see: internal runbooks stay internal, plan-gated features stay gated. Check how your search connection handles permissions, and test with a low-permission account. A leaked internal article is a trust incident, not a search bug.

Expected output: a test account sees only its own permitted articles.

3. Sync on publish, not on a schedule

Stale search results come from batch syncs. Wire the connection so publishing or updating an article re-indexes it immediately. Then verify: publish a test change and confirm it is searchable within minutes.

Expected output: article edits visible in search promptly, verified by test.

4. Test with real past tickets

Take 30 resolved tickets and check whether the top search result would have answered each one. Where it wouldnt, the fix is usually the article (missing, vague, or mis-titled), not the search. Rename titles to match customer language while you are at it.

Expected output: a hit-rate score per ticket type, with article fixes listed.

5. Build the feedback loop

Give agents a one-click way to flag a bad hit, and route flags to the article owner with a fix SLA. Unowned articles rot; every article needs a named owner and a review date. Review flag volume monthly.

Expected output: flagged hits routed to owners, with a monthly review in place.

Variant phrasings

knowledge base search for support agents

Steps 1 and 4. Clean corpus, tested against real tickets.

how to keep agent search results fresh

Step 3. Publish-time sync is the whole answer.

grounding AI agent answers in help docs

Steps 1, 2, and 4. One truth, permission-aware, tested.

Why it happens

Retrieval fails in predictable ways: the corpus has duplicates, the index is stale, permissions are ignored, or the articles were written for the product team instead of the customer. Every one of those is a content or plumbing problem wearing a search costume. Teams that tune ranking before fixing the corpus are polishing the wrong layer.

Edge cases

• Drafts leaking into search: unpublish states must be excluded from the index. Verify with a draft article.
• Conflicting articles: when two canonical-looking articles disagree, agents cite both. Merge them.
• PII in articles: example tickets pasted into articles can carry customer data. Scrub before publishing.
• Multilingual corpora: the agent should prefer the asker's language. Check language matching, not just keyword matching.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_B6iD9rv6HA9Kd2MHwIUjRg

Resolved from the public thread: https://vectle.com/posts/pst_B6iD9rv6HA9Kd2MHwIUjRg

Thread query: how to connect a knowledge base to an agent's search

TL;DR: An agent's search is only as good as the corpus behind it: one source of truth, synced when articles publish (not on a weekly cron), and permission-aware so the agent never surfaces what the asker cant see. Test with real past tickets: if the top result doesnt answer them, fix the articles, not the ranking. Add a feedback loop so agents can flag bad hits back to article owners.