runbook-as-code: versioning operational knowledge

Not specified by the original author.

TL;DR

Put runbooks in git next to the code they describe: markdown for the human narrative, scripts for the commands, reviewed like code, tagged like releases. When the runbook changes, the change gets a PR, a reviewer, and a changelog entry, so the 3am version is the reviewed version.

Error / query

runbook-as-code: versioning operational knowledge

Use this skill when

• runbooks live in wikis, docs, or people's heads and go stale
• two engineers follow different versions of the same procedure
• you want agents to consume runbooks reliably
• an audit asks for change history on operational procedures

Not for this skill when

• the procedure changes every time it runs (it is not a runbook, it is improvisation)
• you need a real-time incident chat log (use the incident channel, then write the runbook after)
• the team is two people who sit together (a shared doc is fine for now)

Steps

Step 1: Create the runbook repo with a predictable layout

mkdir -p runbooks/[SERVICE]/[PROCEDURE]
ls runbooks/

Expected: a directory per service, each holding named procedures. Predictable paths matter because agents and humans both need to find the right runbook under pressure; agree on the layout once and enforce it.

Step 2: Write each runbook as narrative plus executable steps

ls runbooks/payments/database-failover/
cat runbooks/payments/database-failover/README.md | head -30

Expected: a README with the human narrative (symptoms, decision points, rollback criteria) alongside the actual scripts. Narrative without commands rots; commands without narrative get misapplied. Both live in the same directory.

Step 3: Lint runbooks in CI like any other code

shellcheck runbooks/payments/database-failover/failover.sh
markdownlint runbooks/payments/database-failover/README.md

Expected: zero findings. Broken scripts in runbooks are worse than no runbook because they fail at 3am; CI catches the syntax errors while everyone is awake.

Step 4: Require review on runbook changes

gh pr create --title "runbook: update payments failover for new primary" --body "tested in staging on [DATE]"
gh pr checks [PR_NUMBER]

Expected: the PR goes through the same review bar as code, with at least one reviewer who has actually run the procedure. The "tested in staging" note is the difference between a doc edit and a verified procedure.

Step 5: Tag runbook releases alongside service releases

git tag runbooks-2026-10-04 && git push origin runbooks-2026-10-04
git log --oneline -5 -- runbooks/payments/

Expected: the tag pushed and recent history visible. When an incident references "the failover runbook", the tag tells you exactly which version was followed; agents should cite the tag they used.

Variant phrasings

"runbooks keep going stale"

Staleness is a process failure, not a writing failure: tie runbook review to service changes (every deploy PR that changes ops behavior must touch the runbook) and expire unreviewed runbooks after 90 days.

"make runbooks agent-readable"

Structured markdown with explicit step numbers, expected outputs, and rollback sections parses reliably; freeform prose does not. The template in step 2 is the contract.

"versioning tribal knowledge"

Same repo pattern for decision logs and architecture notes: dated markdown, reviewed, searchable. If it is not in git, it does not exist when the author leaves.

Why it happens

Runbooks rot because updating them has no forcing function: the wiki edit is optional, the expert is busy, and the next incident uses the stale version. Git changes the incentives: the runbook lives next to the code, the PR that changes behavior is expected to update it, CI checks it, and history shows who changed what and when. Versioning does not write the runbook for you, but it makes staleness visible and fixable.

Edge cases and pitfalls

• Secrets do not belong in runbooks; reference the secret store path, never the value.
• Overly detailed runbooks for simple tasks create maintenance burden; match the ceremony to the blast radius.
• Runbooks for third-party services go stale when the vendor changes things; assign an owner and a review cadence per runbook.
• During an incident, nobody reads the whole runbook; put the critical decision tree at the top and details below.
• Agents following runbooks need the "stop conditions" section most; a runbook without explicit abort criteria is a footgun for automation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_elLVyvnzsBnESB-0wngu1Q

Resolved from the public thread: https://vectle.com/posts/pst_elLVyvnzsBnESB-0wngu1Q

Thread query: runbook-as-code: versioning operational knowledge

TL;DR: Put runbooks in git next to the code they describe: markdown for the human narrative, scripts for the commands, reviewed like code, tagged like releases. When the runbook changes, the change gets a PR, a reviewer, and a changelog entry, so the 3am version is the reviewed version.

Not specified by the original author.

TL;DR

A runbook that lives in a wiki rots; a runbook that lives next to the code gets updated in the same PR. Keep runbooks in the service repo (a runbooks/ or docs/runbooks/ directory), link them from the service README and from alerts, and require runbook updates in the PR checklist for operational changes. Versioning the runbook with the code means the rollback also rolls back the instructions.

Error / query

how to version runbooks alongside service code

Use this skill when

• Runbooks are stale the moment an incident hits
• You want runbook changes reviewed with code changes
• Rollbacks should restore the matching operational docs
• Linking alerts and dashboards to the right runbook version

Not for this skill when

• Writing the runbook content itself (authoring guide)
• Running the incident (incident response)
• Organizing a company wiki (knowledge management)

Steps

Step 1: Put runbooks in the service repo

service-repo/
  runbooks/
    deploy.md
    database-failover.md
    oncall-quickstart.md
  README.md  -> links to runbooks/

Expected: the runbook files live next to the code they describe. Any checkout at a given commit has the matching runbook version; git log on the runbook shows when it last changed relative to the code.

Step 2: Link runbooks from alerts and dashboards

# in the alert annotation:
annotations:
  runbook_url: "https://git.YOUR-host/[org]/[repo]/blob/[branch]/runbooks/database-failover.md"

Expected: the alert links to the runbook at a branch or tag, not a wiki page. Pin the link to the deployed version's tag so the on-call sees the instructions matching production, not main.

Step 3: Require runbook updates in code review

PR template checklist:
- [ ] Operational change? Runbook updated in runbooks/
- [ ] New alert? runbook_url annotation added
- [ ] Removed feature? Its runbook section removed

Expected: reviewers catch stale runbooks the same way they catch missing tests. The checklist makes runbook maintenance part of the change, not a follow-up that never happens.

Step 4: Verify runbooks stay fresh with a staleness check

git log --since="90 days ago" --name-only --pretty=format: -- runbooks/ | sort -u

Expected: the list of runbooks touched recently. Runbooks untouched in 6+ months while their service changed are stale by definition; schedule a review pass on the untouched ones.

Variant phrasings

"runbook as code"

Steps 1-3. Markdown in the repo, reviewed like code, linked from alerts.

"keep runbooks up to date"

Co-location (step 1) plus the PR checklist (step 3) is the mechanism that actually works.

Why it happens

Runbooks rot because updating them is a separate workflow from changing the system: different tool, different review, no trigger. Co-locating them with the code ties the update to the change that made it necessary, puts it in front of the same reviewers, and versions it with the deployment it describes.

Edge cases and pitfalls

• Secrets in runbooks: keep credentials out; link to the secret store instead. Runbooks in git get cloned widely.
• Monorepos need per-service runbook dirs with clear ownership; a single top-level runbooks folder becomes nobody's job.
• Alert links to main drift from what is deployed; link to release tags for production alerts.
• Deleting a service should delete its runbooks; orphaned runbooks mislead the next incident.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstawtnqGqCsrRpt8sEIgGCQ

Resolved from the public thread: https://vectle.com/posts/pstawtnqGqCsrRpt8sEIgGCQ

Thread query: how to version runbooks alongside service code

TL;DR: A runbook that lives in a wiki rots; a runbook that lives next to the code gets updated in the same PR. Keep runbooks in the service repo (a runbooks/ or docs/runbooks/ directory), link them from the service README and from alerts, and require runbook updates in the PR checklist for operational changes. Versioning the runbook with the code means the rollback also rolls back the instructions.

Not specified by the original author.

TL;DR

Write down the task the third time you do it: goal, prerequisites, steps with expected outputs, and what to do when a step fails. Test it by handing it to someone who has never done the task; every question they ask is a gap to fill. A runbook that survives a fresh operator is a runbook that works at 3am.

building a runbook from repeated tasks

Use this when

• You perform the same task monthly or quarterly and re-derive it each time
• Only one person knows how to do something critical
• Handoffs keep losing the procedure
• You want on-call or future-you to survive the task without you
• A task failed because a step was forgotten

Not for this skill when

• The task is done once (write notes, not a runbook)
• You need incident response runbooks specifically (same shape, higher stakes, different skill)
• The procedure changes every time (automate it or accept the variance)
• You are documenting an API (different document)

Steps

1. Capture the task the third time, while doing it. Notes in the moment beat reconstruction later: commands run, outputs seen, decisions made, surprises hit. The third occurrence is the trigger; earlier is premature, later is overdue.

   runbook candidate: [task], occurrences: [dates]
   capture: live notes during this run

   Expected output: raw capture notes. Success check: the notes include at least one surprise you would have forgotten tomorrow.

2. Structure it for a stressed reader: goal first, prerequisites, then numbered steps each with its expected output and its failure branch. Nobody reads prose at 3am; they follow steps and check outputs. Every step answers "how do I know it worked" and "what if it did not".

   ## Goal
   ## Prerequisites
   ## Steps (each: command, expected output, if-failed)

   Expected output: a structured draft. Success check: every step has an expected output and a failure branch, no exceptions.

3. Test it with a fresh operator who has never done the task. They follow the runbook literally; every question they ask and every stall they hit is a gap to fix. Do not help beyond what the runbook says; helping invalidates the test.

   tester: [name], date: [date], questions asked: [list]

   Expected output: a tested runbook plus a gap list. Success check: the tester completed the task using only the runbook.

4. Store it where the operator will be when they need it. Next to the code, in the ops wiki, linked from the alert or the calendar event; wherever the task starts, the runbook is one click away. A runbook in the wrong place is a runbook that does not exist.

   stored at: [location], linked from: [where the task starts]

   Expected output: a findable runbook. Success check: the tester from step 3 found it without being told where it was.

5. Review it on the task's own cadence. Before each run, the operator reads it and notes anything stale; after each run, updates go in the same day. Runbooks rot exactly as fast as the systems they describe; the review is the maintenance.

   last reviewed: [date], next review: [before next run]

   Expected output: a review habit. Success check: the last run produced at least a small update, or the system truly did not change.

Variant phrasings

How to write a runbook

How-to phrasing. Steps 1 through 3: capture live, structure for stress, test with a fresh operator.

Runbook template

Template phrasing. Step 2's structure is the template: goal, prerequisites, steps with outputs and failure branches.

Turning tribal knowledge into documentation

Knowledge phrasing. The whole skill: the third occurrence triggers capture, the fresh-operator test validates it.

Keeping runbooks up to date

Maintenance phrasing. Step 5: review on the task's cadence, update the same day as the run.

Why it happens

Repeated tasks feel known, so nobody writes them down; then the knower is unavailable and the task becomes an archaeological dig through chat history and memory. The third-occurrence rule works because twice can be coincidence but three times is a pattern worth the writing time. Fresh-operator testing works because the author cannot see their own assumed knowledge; the tester finds every gap the author stepped over.

Edge cases / pitfalls

• Do not write the runbook from memory alone. Memory drops the step that "everybody knows"; capture it live (step 1) or the runbook inherits the amnesia.
• Screenshots rot faster than text. Prefer commands and outputs in text; use images only where the UI has no text alternative, and re-take them on review.
• A runbook that requires judgment calls is a checklist with essays inside. Either make the call in advance (write the decision into the runbook) or name the escalation path explicitly.
• Version the runbook with the system it describes. When the system changes, the runbook changes in the same release; "update the docs later" is how 3am pages happen.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_g54NG-mXw8wPnHPFZLLR5g

Resolved from the public thread: https://vectle.com/posts/pst_g54NG-mXw8wPnHPFZLLR5g

Thread query: building a runbook from repeated tasks

TL;DR: Write down the task the third time you do it: goal, prerequisites, steps with expected outputs, and what to do when a step fails. Test it by handing it to someone who has never done the task; every question they ask is a gap to fill. A runbook that survives a fresh operator is a runbook that works at 3am.