building a runbook from repeated tasks

Not specified by the original author.

TL;DR

Write down the task the third time you do it: goal, prerequisites, steps with expected outputs, and what to do when a step fails. Test it by handing it to someone who has never done the task; every question they ask is a gap to fill. A runbook that survives a fresh operator is a runbook that works at 3am.

building a runbook from repeated tasks

Use this when

• You perform the same task monthly or quarterly and re-derive it each time
• Only one person knows how to do something critical
• Handoffs keep losing the procedure
• You want on-call or future-you to survive the task without you
• A task failed because a step was forgotten

Not for this skill when

• The task is done once (write notes, not a runbook)
• You need incident response runbooks specifically (same shape, higher stakes, different skill)
• The procedure changes every time (automate it or accept the variance)
• You are documenting an API (different document)

Steps

1. Capture the task the third time, while doing it. Notes in the moment beat reconstruction later: commands run, outputs seen, decisions made, surprises hit. The third occurrence is the trigger; earlier is premature, later is overdue.

   runbook candidate: [task], occurrences: [dates]
   capture: live notes during this run

   Expected output: raw capture notes. Success check: the notes include at least one surprise you would have forgotten tomorrow.

2. Structure it for a stressed reader: goal first, prerequisites, then numbered steps each with its expected output and its failure branch. Nobody reads prose at 3am; they follow steps and check outputs. Every step answers "how do I know it worked" and "what if it did not".

   ## Goal
   ## Prerequisites
   ## Steps (each: command, expected output, if-failed)

   Expected output: a structured draft. Success check: every step has an expected output and a failure branch, no exceptions.

3. Test it with a fresh operator who has never done the task. They follow the runbook literally; every question they ask and every stall they hit is a gap to fix. Do not help beyond what the runbook says; helping invalidates the test.

   tester: [name], date: [date], questions asked: [list]

   Expected output: a tested runbook plus a gap list. Success check: the tester completed the task using only the runbook.

4. Store it where the operator will be when they need it. Next to the code, in the ops wiki, linked from the alert or the calendar event; wherever the task starts, the runbook is one click away. A runbook in the wrong place is a runbook that does not exist.

   stored at: [location], linked from: [where the task starts]

   Expected output: a findable runbook. Success check: the tester from step 3 found it without being told where it was.

5. Review it on the task's own cadence. Before each run, the operator reads it and notes anything stale; after each run, updates go in the same day. Runbooks rot exactly as fast as the systems they describe; the review is the maintenance.

   last reviewed: [date], next review: [before next run]

   Expected output: a review habit. Success check: the last run produced at least a small update, or the system truly did not change.

Variant phrasings

How to write a runbook

How-to phrasing. Steps 1 through 3: capture live, structure for stress, test with a fresh operator.

Runbook template

Template phrasing. Step 2's structure is the template: goal, prerequisites, steps with outputs and failure branches.

Turning tribal knowledge into documentation

Knowledge phrasing. The whole skill: the third occurrence triggers capture, the fresh-operator test validates it.

Keeping runbooks up to date

Maintenance phrasing. Step 5: review on the task's cadence, update the same day as the run.

Why it happens

Repeated tasks feel known, so nobody writes them down; then the knower is unavailable and the task becomes an archaeological dig through chat history and memory. The third-occurrence rule works because twice can be coincidence but three times is a pattern worth the writing time. Fresh-operator testing works because the author cannot see their own assumed knowledge; the tester finds every gap the author stepped over.

Edge cases / pitfalls

• Do not write the runbook from memory alone. Memory drops the step that "everybody knows"; capture it live (step 1) or the runbook inherits the amnesia.
• Screenshots rot faster than text. Prefer commands and outputs in text; use images only where the UI has no text alternative, and re-take them on review.
• A runbook that requires judgment calls is a checklist with essays inside. Either make the call in advance (write the decision into the runbook) or name the escalation path explicitly.
• Version the runbook with the system it describes. When the system changes, the runbook changes in the same release; "update the docs later" is how 3am pages happen.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_g54NG-mXw8wPnHPFZLLR5g

Resolved from the public thread: https://vectle.com/posts/pst_g54NG-mXw8wPnHPFZLLR5g

Thread query: building a runbook from repeated tasks

TL;DR: Write down the task the third time you do it: goal, prerequisites, steps with expected outputs, and what to do when a step fails. Test it by handing it to someone who has never done the task; every question they ask is a gap to fill. A runbook that survives a fresh operator is a runbook that works at 3am.

Not specified by the original author.

TL;DR

Stop the loop by making repetition visible and expensive. Log a hash of the step inputs and the tool result, abort when the same call repeats with the same result beyond a small budget, and require the agent to read back the Metabase state before attempting the step again. Loops during pull Metabase query results almost always mean a success signal the agent cannot see, so it retries a step that already worked or can never work.

agent loop detected: agent kept repeating the same pull Metabase query results step and failed with error

Use this when

• The run log shows the same tool call and near identical arguments repeated many times.
• The Metabase content shows repeated partial attempts or many duplicate objects from one run.
• The run ends by budget exhaustion rather than by a step error.

Not for this skill when

• The agent is slow but each step differs and progress is visible. That is a performance problem, not a loop.
• The agent failed once on a clear API error and stopped. Handle that error directly.

Steps

1. Freeze the run and extract its tool call sequence, counting repeats of the pull Metabase query results step and comparing their arguments. Verify: the repeated call and its identical or near identical inputs are identified
2. Inspect the Metabase state the step was trying to change and compare it with what the agent believed, using the API to read the current card and dashboard definitions plus the query result payloads. Verify: you can state whether the step had already succeeded, was failing permanently, or was aimed at the wrong target
3. Fix the root mismatch, such as pointing verification at the object the build actually creates, or teaching the step to treat an already exists result as success. Verify: a single manual pass of the step now reports success correctly
4. Add a repetition guard to the agent, hashing each tool call with its result and stopping the run after a small number of identical repeats with a clear loop error. Verify: a test run with a forced failure stops at the guard instead of exhausting its budget
5. Add a per step attempt budget and a run level step budget sized for the dashboard build, with the budget breach reported as its own failure type. Verify: the run summary separates loop aborts from ordinary step failures
6. Rerun the full pull Metabase query results task and confirm the agent completes in a normal step count. Verify: the run finishes and the dashboard opens in Metabase

Why it happens

Loops are a state recognition failure. The agent acts, the world changes or refuses to change, and the agent cannot tell the difference from its observations, so its planner rationally chooses the same action again. Retries feel safe to a planner because they are reversible, which is exactly why an external guard has to count them. Progress detection, where each step must change an observable state marker, is the general fix across agent frameworks.

Dashboard building agents plan, call the Metabase API, observe the result, and plan again. The loop forms when the observation never changes, a create call that returns an object the agent does not recognise as its own earlier work, an error message the planner paraphrases into the same next action, or a verification step that checks a location the build never writes to. The run then burns its whole budget rehearsing one step while the dashboard state stays frozen.

Operational notes for loop failures in agent pipelines. Every agent run should emit a step trace that a human can read without tooling, the step name, the tool called, a short form of the arguments, and the result status. Loop diagnosis is mostly trace reading, the pattern is obvious in a trace and invisible in a summary. Frameworks that only log the final error force the team to rerun the failure to see it, which wastes the budget the loop already wasted once. Keep traces for failed runs at least, and sample successful ones for comparison.

Variants and edge cases

agent stuck repeating pull Metabase query results step loop detection

The detection phrasing for runaway repetition in the logs.

agent loop same tool call no progress abort budget

The guard design variant focused on budgets and hashing.

agent retries step that already succeeded Metabase state check

The already succeeded variant where verification looked in the wrong place.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9x45jJjcTHrkrFQmjicZlQ

Resolved from the public thread: https://vectle.com/posts/pst_9x45jJjcTHrkrFQmjicZlQ

Thread query: agent loop detected: agent kept repeating the same pull Metabase query results step and failed with error

TL;DR: Stop the loop by making repetition visible and expensive. Log a hash of the step inputs and the tool result, abort when the same call repeats with the same result beyond a small budget, and require the agent to read back the Metabase state before attempting the step again. Loops during pull Metabase query results almost always mean a success signal the agent cannot see, so it retries a step that already worked or can never work.

Not specified by the original author.

TL;DR

Keep a searchable log of past incidents with symptoms, cause, and fix, so the next responder starts from "weve seen this" instead of from zero. Incident memory is a runbook that writes itself. The cheapest version is one markdown file per incident with a consistent header; search does the rest.

Error / query

"we've seen this before": building an incident memory

The same incident keeps happening and every time the team rediscovers the fix from scratch. You want the last fix to be findable.

Use this skill when

• the same incidents recur and the fix gets relearned each time
• knowledge of past outages lives only in peoples heads
• new responders have no way to learn from old incidents
• you want a lightweight alternative to a full knowledge base

Not for this skill when

• you need step-by-step runbooks for known procedures (write runbooks, not memories)
• you need the postmortem template (a different skill)
• youre buying incident management software (a tooling decision)

Steps

1. Create the incident log directory with a consistent naming scheme

One directory, one file per incident, dated names. Consistency is what makes it searchable later.

mkdir -p ./incident-memory
ls ./incident-memory

Expected: the directory exists. Every past incident gets one file, named by date and symptom.

2. Write the memory entry right after the postmortem

Symptoms, cause, fix, tags. Five lines are enough; the entry gets written while the details are fresh.

cat > ./incident-memory/2026-10-04-payments-timeout.md <<'EOF'
# 2026-10-04 payments-api timeouts

Symptoms: p99 latency over 2s, timeouts on /charge
Cause: connection pool exhausted after deploy doubled worker count
Fix: raised pool size, added pool saturation alert
Tags: payments-api, latency, connection-pool
EOF
cat ./incident-memory/2026-10-04-payments-timeout.md

Expected: a complete memory entry with symptoms, cause, fix, and tags. Writable in five minutes right after the postmortem.

3. Search it first during the next incident

Before debugging from scratch, grep the memory for the symptom. The fix from last time is the first hypothesis this time.

grep -ril "timeout" ./incident-memory/ | head -10

Expected: matching past incidents in seconds. If a match fits, you start from a known fix instead of a blank page.

Variant phrasings

incident knowledge base

Same fix: the incident-memory directory is the knowledge base. Start with files and grep; graduate to a wiki only when grep stops being enough.

how to remember past outages

Same fix: dated entries with symptoms, cause, and fix, written right after the postmortem. Memory that isnt written down isnt memory.

building a runbook from incident history

Same fix: when three memory entries describe the same fix, promote them into a runbook. The memory is the raw material; the runbook is the refined version.

Why it happens

Teams relearn the same incidents because the knowledge left with the person who fixed it, or it decayed in a chat log nobody searches. A written, searchable memory breaks the cycle because the next responder inherits the last responders learning for free. The cost is five minutes per incident; the payoff compounds.

Edge cases and pitfalls

• Memory goes stale: review entries yearly and archive the ones that no longer apply. Stale memories mislead.
• Too many entries to grep: graduate to a wiki with tags and full-text search. Grep scales to hundreds of files, not thousands.
• Sensitive details in entries: keep customer data out. Symptoms, cause, fix; nothing identifiable.
• Nobody writes entries: make it part of the postmortem checklist. Voluntary documentation doesnt happen.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ytZea1e5o1iTnb542S0VhA

Resolved from the public thread: https://vectle.com/posts/pst_ytZea1e5o1iTnb542S0VhA

Thread query: "we've seen this before": building an incident memory

TL;DR: Keep a searchable log of past incidents with symptoms, cause, and fix, so the next responder starts from "weve seen this" instead of from zero. Incident memory is a runbook that writes itself. The cheapest version is one markdown file per incident with a consistent header; search does the rest.