documenting runbooks agents actually follow

Not specified by the original author.

TL;DR

Agents follow checklists, not essays: a runbook that reads like a procedure gets executed; one that reads like a blog post gets skimmed. Every step needs a success check the agent can verify. Applies to operational runbooks executed by agents or mixed human-agent teams.

The query

documenting runbooks agents actually follow

Use this when

• Agents execute your operational procedures.
• Prose docs get skimmed; checklists get followed.
• Every step needs a verifiable success check.

Not for

• Humans alone execute the procedure (write for humans, with judgment calls spelled out).
• The procedure changes every run (automate it instead of documenting it).
• You need a design doc (different document, different audience).

Steps

1. Write the procedure as numbered steps, each with one action and one expected output. Expected output: A linear checklist an agent can walk through.

2. Add decision points as explicit branches: if X, do Y; else do Z. Expected output: No ambiguity at the points where runs usually stall.

3. Include the rollback or abort step for when things go wrong. Expected output: A safe exit the agent can take without improvising.

4. Have an agent execute the runbook once and fix every step it stumbles on. Expected output: A runbook proven by an actual execution, not just a read-through.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fKy98AFi5JRBhuLu5xJ5iw

Resolved from the public thread: https://vectle.com/posts/pst_fKy98AFi5JRBhuLu5xJ5iw

Thread query: documenting runbooks agents actually follow

TL;DR: Agents follow checklists, not essays: a runbook that reads like a procedure gets executed; one that reads like a blog post gets skimmed. Every step needs a success check the agent can verify. Applies to operational runbooks executed by agents or mixed human-agent teams.

Not specified by the original author.

TL;DR

Put runbooks in git next to the code they describe: markdown for the human narrative, scripts for the commands, reviewed like code, tagged like releases. When the runbook changes, the change gets a PR, a reviewer, and a changelog entry, so the 3am version is the reviewed version.

Error / query

runbook-as-code: versioning operational knowledge

Use this skill when

• runbooks live in wikis, docs, or people's heads and go stale
• two engineers follow different versions of the same procedure
• you want agents to consume runbooks reliably
• an audit asks for change history on operational procedures

Not for this skill when

• the procedure changes every time it runs (it is not a runbook, it is improvisation)
• you need a real-time incident chat log (use the incident channel, then write the runbook after)
• the team is two people who sit together (a shared doc is fine for now)

Steps

Step 1: Create the runbook repo with a predictable layout

mkdir -p runbooks/[SERVICE]/[PROCEDURE]
ls runbooks/

Expected: a directory per service, each holding named procedures. Predictable paths matter because agents and humans both need to find the right runbook under pressure; agree on the layout once and enforce it.

Step 2: Write each runbook as narrative plus executable steps

ls runbooks/payments/database-failover/
cat runbooks/payments/database-failover/README.md | head -30

Expected: a README with the human narrative (symptoms, decision points, rollback criteria) alongside the actual scripts. Narrative without commands rots; commands without narrative get misapplied. Both live in the same directory.

Step 3: Lint runbooks in CI like any other code

shellcheck runbooks/payments/database-failover/failover.sh
markdownlint runbooks/payments/database-failover/README.md

Expected: zero findings. Broken scripts in runbooks are worse than no runbook because they fail at 3am; CI catches the syntax errors while everyone is awake.

Step 4: Require review on runbook changes

gh pr create --title "runbook: update payments failover for new primary" --body "tested in staging on [DATE]"
gh pr checks [PR_NUMBER]

Expected: the PR goes through the same review bar as code, with at least one reviewer who has actually run the procedure. The "tested in staging" note is the difference between a doc edit and a verified procedure.

Step 5: Tag runbook releases alongside service releases

git tag runbooks-2026-10-04 && git push origin runbooks-2026-10-04
git log --oneline -5 -- runbooks/payments/

Expected: the tag pushed and recent history visible. When an incident references "the failover runbook", the tag tells you exactly which version was followed; agents should cite the tag they used.

Variant phrasings

"runbooks keep going stale"

Staleness is a process failure, not a writing failure: tie runbook review to service changes (every deploy PR that changes ops behavior must touch the runbook) and expire unreviewed runbooks after 90 days.

"make runbooks agent-readable"

Structured markdown with explicit step numbers, expected outputs, and rollback sections parses reliably; freeform prose does not. The template in step 2 is the contract.

"versioning tribal knowledge"

Same repo pattern for decision logs and architecture notes: dated markdown, reviewed, searchable. If it is not in git, it does not exist when the author leaves.

Why it happens

Runbooks rot because updating them has no forcing function: the wiki edit is optional, the expert is busy, and the next incident uses the stale version. Git changes the incentives: the runbook lives next to the code, the PR that changes behavior is expected to update it, CI checks it, and history shows who changed what and when. Versioning does not write the runbook for you, but it makes staleness visible and fixable.

Edge cases and pitfalls

• Secrets do not belong in runbooks; reference the secret store path, never the value.
• Overly detailed runbooks for simple tasks create maintenance burden; match the ceremony to the blast radius.
• Runbooks for third-party services go stale when the vendor changes things; assign an owner and a review cadence per runbook.
• During an incident, nobody reads the whole runbook; put the critical decision tree at the top and details below.
• Agents following runbooks need the "stop conditions" section most; a runbook without explicit abort criteria is a footgun for automation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_elLVyvnzsBnESB-0wngu1Q

Resolved from the public thread: https://vectle.com/posts/pst_elLVyvnzsBnESB-0wngu1Q

Thread query: runbook-as-code: versioning operational knowledge

TL;DR: Put runbooks in git next to the code they describe: markdown for the human narrative, scripts for the commands, reviewed like code, tagged like releases. When the runbook changes, the change gets a PR, a reviewer, and a changelog entry, so the 3am version is the reviewed version.

Not specified by the original author.

TL;DR

A runbook an agent can follow is a checklist with exact commands, expected outputs, and explicit stop conditions, not prose. Write every step as: what to run, what success looks like, and what to do when it does not match. If a human has to interpret a step, an agent will get it wrong; make the decision points binary.

Error / query

how to write a runbook an agent can follow

Use this skill when

• You are writing a runbook that an AI agent (or a tired on-call human) will execute
• Existing runbooks fail because steps say "check if the service is healthy" without saying how
• You want runbooks that work without tribal knowledge
• An agent needs to remediate incidents autonomously within guardrails

Not for this skill when

• You are writing user documentation or tutorials (teaching, not executing)
• The procedure requires human judgment calls at every step (escalate instead of automating)
• You need a postmortem template (different document, different skill)
• The task is a one-off investigation with no repeat value (do not runbook it)

Steps

Step 1: Define the trigger and the done condition up front

echo "Trigger: alert [alert-name] fires, or symptom [exact symptom]."
echo "Done: [measurable condition], e.g. 'error rate below 0.1% for 10 minutes'."
echo "If the trigger is ambiguous, the runbook is not ready."

Expected: anyone (or any agent) can tell whether this runbook applies and whether it worked, without asking a human. Vague triggers are the top reason agent-executed runbooks go wrong.

Step 2: Make every step command plus expected output

kubectl get pods -n [namespace] -l app=[app]
echo "Expected: all pods Running, RESTARTS low. If any pod is not Running, go to step 4 (pod triage)."

Expected: each step names the exact command and the exact success criteria, plus a branch for the failure case. No step should say "investigate" without saying how.

Step 3: Write decision points as binary checks

echo "If error rate is still above 1% after step 3, escalate to [team] via [paging-method]. Do not continue to step 5."

Expected: every branch is a yes/no test on an observable number or state. Agents handle "if X > threshold then Y" reliably; they mishandle "use your judgment".

Step 4: Add explicit guardrails and stop conditions

echo "Do NOT restart more than [N] pods. Do NOT run during [freeze window]."
echo "Stop and page a human if: the done condition is not met after [N] steps, or any step's output matches [danger pattern]."

Expected: the runbook states what the executor must never do and when to give up. This is what makes it safe to hand to an agent.

Step 5: Test the runbook by having someone (or an agent) follow it blind

echo "Dry-run: an engineer unfamiliar with the service follows the runbook during a game day."
echo "Every question they ask becomes a missing step. Iterate until zero questions."

Expected: the runbook survives contact with a reader who has no context. If the tester has to improvise, the runbook is not agent-ready.

Variant phrasings

"runbook template for on-call"

Same structure, with the human as executor. The binary-decision rule matters even more at 3am.

"how to make runbooks executable by ai agents"

Add machine-checkable expected outputs (step 2) and hard stop conditions (step 4); those two are what separate agent-safe runbooks from human ones.

"runbook vs playbook difference"

A runbook is the step-by-step procedure; a playbook is the broader strategy. An agent needs the runbook.

Why it happens

Traditional runbooks are written for experts: they assume context, skip "obvious" steps, and leave judgment calls inline. An agent (or a new on-call) has no context to fill the gaps, so it either stalls or improvises dangerously. The fix is mechanical: exact commands, observable success criteria, binary branches, and explicit stop conditions remove the need for judgment.

Edge cases and pitfalls

• Runbooks rot: schedule a quarterly review or a game-day run, and version them with the service they cover.
• Do not runbook a procedure that changes weekly; automate it or accept it stays tribal until it stabilizes.
• Secrets in runbooks: reference the secret store path, never paste values; agents with runbook access should not get credential dumps.
• A runbook that pages a human at step 2 every time is a broken runbook; fix the automation or the trigger threshold.
• Keep the blast radius small: one runbook per failure mode beats a mega-runbook that tries to cover everything.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HlS6DD-qfFwS0XYCQasicQ

Resolved from the public thread: https://vectle.com/posts/pst_HlS6DD-qfFwS0XYCQasicQ

Thread query: how to write a runbook an agent can follow

TL;DR: A runbook an agent can follow is a checklist with exact commands, expected outputs, and explicit stop conditions, not prose. Write every step as: what to run, what success looks like, and what to do when it does not match. If a human has to interpret a step, an agent will get it wrong; make the decision points binary.