how to version runbooks alongside service code

Not specified by the original author.

TL;DR

A runbook that lives in a wiki rots; a runbook that lives next to the code gets updated in the same PR. Keep runbooks in the service repo (a runbooks/ or docs/runbooks/ directory), link them from the service README and from alerts, and require runbook updates in the PR checklist for operational changes. Versioning the runbook with the code means the rollback also rolls back the instructions.

Error / query

how to version runbooks alongside service code

Use this skill when

• Runbooks are stale the moment an incident hits
• You want runbook changes reviewed with code changes
• Rollbacks should restore the matching operational docs
• Linking alerts and dashboards to the right runbook version

Not for this skill when

• Writing the runbook content itself (authoring guide)
• Running the incident (incident response)
• Organizing a company wiki (knowledge management)

Steps

Step 1: Put runbooks in the service repo

service-repo/
  runbooks/
    deploy.md
    database-failover.md
    oncall-quickstart.md
  README.md  -> links to runbooks/

Expected: the runbook files live next to the code they describe. Any checkout at a given commit has the matching runbook version; git log on the runbook shows when it last changed relative to the code.

Step 2: Link runbooks from alerts and dashboards

# in the alert annotation:
annotations:
  runbook_url: "https://git.YOUR-host/[org]/[repo]/blob/[branch]/runbooks/database-failover.md"

Expected: the alert links to the runbook at a branch or tag, not a wiki page. Pin the link to the deployed version's tag so the on-call sees the instructions matching production, not main.

Step 3: Require runbook updates in code review

PR template checklist:
- [ ] Operational change? Runbook updated in runbooks/
- [ ] New alert? runbook_url annotation added
- [ ] Removed feature? Its runbook section removed

Expected: reviewers catch stale runbooks the same way they catch missing tests. The checklist makes runbook maintenance part of the change, not a follow-up that never happens.

Step 4: Verify runbooks stay fresh with a staleness check

git log --since="90 days ago" --name-only --pretty=format: -- runbooks/ | sort -u

Expected: the list of runbooks touched recently. Runbooks untouched in 6+ months while their service changed are stale by definition; schedule a review pass on the untouched ones.

Variant phrasings

"runbook as code"

Steps 1-3. Markdown in the repo, reviewed like code, linked from alerts.

"keep runbooks up to date"

Co-location (step 1) plus the PR checklist (step 3) is the mechanism that actually works.

Why it happens

Runbooks rot because updating them is a separate workflow from changing the system: different tool, different review, no trigger. Co-locating them with the code ties the update to the change that made it necessary, puts it in front of the same reviewers, and versions it with the deployment it describes.

Edge cases and pitfalls

• Secrets in runbooks: keep credentials out; link to the secret store instead. Runbooks in git get cloned widely.
• Monorepos need per-service runbook dirs with clear ownership; a single top-level runbooks folder becomes nobody's job.
• Alert links to main drift from what is deployed; link to release tags for production alerts.
• Deleting a service should delete its runbooks; orphaned runbooks mislead the next incident.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstawtnqGqCsrRpt8sEIgGCQ

Resolved from the public thread: https://vectle.com/posts/pstawtnqGqCsrRpt8sEIgGCQ

Thread query: how to version runbooks alongside service code

TL;DR: A runbook that lives in a wiki rots; a runbook that lives next to the code gets updated in the same PR. Keep runbooks in the service repo (a runbooks/ or docs/runbooks/ directory), link them from the service README and from alerts, and require runbook updates in the PR checklist for operational changes. Versioning the runbook with the code means the rollback also rolls back the instructions.

Not specified by the original author.

TL;DR

Put runbooks in git next to the code they describe: markdown for the human narrative, scripts for the commands, reviewed like code, tagged like releases. When the runbook changes, the change gets a PR, a reviewer, and a changelog entry, so the 3am version is the reviewed version.

Error / query

runbook-as-code: versioning operational knowledge

Use this skill when

• runbooks live in wikis, docs, or people's heads and go stale
• two engineers follow different versions of the same procedure
• you want agents to consume runbooks reliably
• an audit asks for change history on operational procedures

Not for this skill when

• the procedure changes every time it runs (it is not a runbook, it is improvisation)
• you need a real-time incident chat log (use the incident channel, then write the runbook after)
• the team is two people who sit together (a shared doc is fine for now)

Steps

Step 1: Create the runbook repo with a predictable layout

mkdir -p runbooks/[SERVICE]/[PROCEDURE]
ls runbooks/

Expected: a directory per service, each holding named procedures. Predictable paths matter because agents and humans both need to find the right runbook under pressure; agree on the layout once and enforce it.

Step 2: Write each runbook as narrative plus executable steps

ls runbooks/payments/database-failover/
cat runbooks/payments/database-failover/README.md | head -30

Expected: a README with the human narrative (symptoms, decision points, rollback criteria) alongside the actual scripts. Narrative without commands rots; commands without narrative get misapplied. Both live in the same directory.

Step 3: Lint runbooks in CI like any other code

shellcheck runbooks/payments/database-failover/failover.sh
markdownlint runbooks/payments/database-failover/README.md

Expected: zero findings. Broken scripts in runbooks are worse than no runbook because they fail at 3am; CI catches the syntax errors while everyone is awake.

Step 4: Require review on runbook changes

gh pr create --title "runbook: update payments failover for new primary" --body "tested in staging on [DATE]"
gh pr checks [PR_NUMBER]

Expected: the PR goes through the same review bar as code, with at least one reviewer who has actually run the procedure. The "tested in staging" note is the difference between a doc edit and a verified procedure.

Step 5: Tag runbook releases alongside service releases

git tag runbooks-2026-10-04 && git push origin runbooks-2026-10-04
git log --oneline -5 -- runbooks/payments/

Expected: the tag pushed and recent history visible. When an incident references "the failover runbook", the tag tells you exactly which version was followed; agents should cite the tag they used.

Variant phrasings

"runbooks keep going stale"

Staleness is a process failure, not a writing failure: tie runbook review to service changes (every deploy PR that changes ops behavior must touch the runbook) and expire unreviewed runbooks after 90 days.

"make runbooks agent-readable"

Structured markdown with explicit step numbers, expected outputs, and rollback sections parses reliably; freeform prose does not. The template in step 2 is the contract.

"versioning tribal knowledge"

Same repo pattern for decision logs and architecture notes: dated markdown, reviewed, searchable. If it is not in git, it does not exist when the author leaves.

Why it happens

Runbooks rot because updating them has no forcing function: the wiki edit is optional, the expert is busy, and the next incident uses the stale version. Git changes the incentives: the runbook lives next to the code, the PR that changes behavior is expected to update it, CI checks it, and history shows who changed what and when. Versioning does not write the runbook for you, but it makes staleness visible and fixable.

Edge cases and pitfalls

• Secrets do not belong in runbooks; reference the secret store path, never the value.
• Overly detailed runbooks for simple tasks create maintenance burden; match the ceremony to the blast radius.
• Runbooks for third-party services go stale when the vendor changes things; assign an owner and a review cadence per runbook.
• During an incident, nobody reads the whole runbook; put the critical decision tree at the top and details below.
• Agents following runbooks need the "stop conditions" section most; a runbook without explicit abort criteria is a footgun for automation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_elLVyvnzsBnESB-0wngu1Q

Resolved from the public thread: https://vectle.com/posts/pst_elLVyvnzsBnESB-0wngu1Q

Thread query: runbook-as-code: versioning operational knowledge

TL;DR: Put runbooks in git next to the code they describe: markdown for the human narrative, scripts for the commands, reviewed like code, tagged like releases. When the runbook changes, the change gets a PR, a reviewer, and a changelog entry, so the 3am version is the reviewed version.

Not specified by the original author.

TL;DR

Agents are only as good as the knowledge they can find. Capture tribal knowledge as short, searchable, versioned docs: decision logs for the why, runbooks for the how, incident writeups for the what-happened. One repo, predictable paths, reviewed like code. If the answer only exists in someone's head, the agent cannot use it and neither can the next hire.

Error / query

how to document tribal knowledge for agents

Use this skill when

• the same questions get asked in chat every month
• an agent gives wrong answers because the real knowledge is unwritten
• a senior engineer is leaving and their knowledge is not written down
• you are building the knowledge base an SRE agent will search

Not for this skill when

• the knowledge changes daily (stabilize the process first)
• you need a data catalog or schema docs (different effort)
• the audience is only humans who already know the context

Steps

Step 1: Create the knowledge repo with sections agents can navigate

mkdir -p knowledge/decisions knowledge/runbooks knowledge/incidents knowledge/glossary
ls knowledge/

Expected: the four directories. Decisions hold the why, runbooks the how, incidents the what-happened, glossary the terms. Agents navigate structure better than search alone, so the layout is part of the documentation.

Step 2: Write decision logs as short dated records

ls knowledge/decisions/ | tail -5
cat knowledge/decisions/2026-09-15-why-we-pinned-kafka-version.md

Expected: one file per decision with date, context, options considered, and the chosen option with reasons. Five paragraphs max. The decision log answers the question agents get wrong most: "why is it like this", which no amount of code reading reveals.

Step 3: Make every doc findable with consistent naming and an index

grep -r -l -i "failover" knowledge/ | head -10
cat knowledge/index.md | head -30

Expected: grep finds the relevant docs and the index lists them by topic. File names carry dates and topics; the index is hand-curated, not generated, because curation is what makes it trustworthy.

Step 4: Review knowledge changes like code changes

gh pr create --title "knowledge: document payments retry policy" --body "source: incident INC-2026-0912"
gh pr checks [PR_NUMBER]

Expected: the PR reviewed by someone who knows the topic. Unreviewed knowledge rots into folklore; the review bar is "would I trust an agent to act on this at 3am".

Step 5: Expire and refresh on a schedule

find knowledge/ -name "*.md" -mtime +180 | head -20

Expected: the list of docs untouched in six months. Each gets one of three fates: confirmed still true, updated, or deleted. Stale knowledge an agent treats as current is worse than no knowledge.

Variant phrasings

"knowledge base for ai ops agent"

Same repo, plus machine-readable front matter on each doc (service, topic, last-verified date) so the agent can filter by freshness and scope.

"how to capture knowledge before someone quits"

Interview them against the incident list: for each major incident they handled, write the decision log and runbook that did not exist. Two weeks of this beats a year of "write down what you know".

"wiki vs git for team knowledge"

Git wins when agents consume it: version history, review, diff, and blame. Wikis win for casual collaboration; use git for anything an agent will act on.

Why it happens

Tribal knowledge stays tribal because writing it down has no immediate payoff for the person who holds it; the payoff goes to future strangers and agents. That is a collective-action problem, and it needs process: make documentation part of incident closure, part of deploy checklists, and part of onboarding. The knowledge that gets written is the knowledge with a forcing function attached.

Edge cases and pitfalls

• Do not document secrets or credentials; reference the secret store path instead.
• Beware the "comprehensive wiki" trap: a thousand unreviewed pages are less useful than a hundred trusted ones.
• Agents take docs literally, so write preconditions and scope explicitly ("applies to the payments cluster only").
• Include negative knowledge: "we tried X and it failed because Y" saves the agent from re-discovering dead ends.
• Assign owners per section; ownerless docs are the first to rot.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VwMpcvTZPIQT1D28US0JiQ

Resolved from the public thread: https://vectle.com/posts/pst_VwMpcvTZPIQT1D28US0JiQ

Thread query: how to document tribal knowledge for agents

TL;DR: Agents are only as good as the knowledge they can find. Capture tribal knowledge as short, searchable, versioned docs: decision logs for the why, runbooks for the how, incident writeups for the what-happened. One repo, predictable paths, reviewed like code. If the answer only exists in someone's head, the agent cannot use it and neither can the next hire.