building skills from your own operational logs

TL;DR

Your logs already contain your best skills: every incident, workaround, and repeated question is a draft waiting to be generalized. Mine the logs monthly for patterns that recur, extract the procedure that worked, strip anything private, and publish it as a skill. The team that writes down what it learned stops paying tuition on the same lesson twice.

building skills from your own operational logs

Use this when

• The same problems get solved from scratch every few months
• Incident chats contain procedures nobody wrote down
• You want a skill pipeline fed by real experience
• New team members keep asking questions the logs already answer
• You are building a knowledge base and need source material

Not for this skill when

• You need log aggregation or search tooling (infrastructure, different skill)
• The logs contain secrets or customer data that cannot be sanitized (do not publish those)
• The knowledge is trivial or already documented (mine for gaps, not volume)
• You want auto-generated docs with no human review (the review is the quality gate)

Steps

1. Pick a log source and read it like an anthropologist, monthly. Incident channels, support threads, deploy logs, cron failure mails. Look for the third occurrence of anything: twice is coincidence, three times is a candidate skill.

   source: [which log], cadence: monthly
   candidates: [pattern seen 3+ times]

Expected output: a candidate list per source. Success check: at least two candidates per review, or you are reading too narrowly.

1. For each candidate, extract the procedure that actually worked. Not the theory, the commands run, the order tried, the thing that finally fixed it. Logs are honest about what worked in a way documentation rarely is; capture that honesty.

   [ ] the working procedure reconstructed from the log
   [ ] dead ends noted (so readers skip them)
   [ ] the fix verified as the actual resolution, not a guess

Expected output: a procedure per candidate. Success check: someone who was not in the incident can follow it.

1. Generalize one level up and strip everything private. Replace your hostnames, keys, customer names, and internal paths with placeholders; lift the procedure from "our deploy" to "a deploy like ours". The skill should help a stranger, not describe your Tuesday.

   [ ] no secrets, hostnames, or customer identifiers remain
   [ ] procedure reads as general advice, not an incident report

Expected output: a sanitized, generalized draft. Success check: it passes your privacy review with zero findings.

1. Validate the draft against reality before publishing. Run the procedure fresh, or have someone uninvolved run it; logs describe what worked once, and once can be luck. Validation turns a war story into a skill.

   [ ] procedure re-run successfully by someone not in the original incident

Expected output: a validated draft. Success check: the re-run succeeded without asking the original solver for help.

1. Publish with the log excerpt as evidence, and link back. The skill cites the operational history ("seen 4 times in Q3"); the log links to the skill for next time. The loop closes when the next occurrence gets answered with the skill link instead of a fresh investigation.

   [ ] skill published, evidence linked
   [ ] next occurrence answered with the link: [date or pending]

Expected output: a closed loop. Success check: a repeat incident gets the skill link as its first response.

Variant phrasings

Turning incidents into documentation

Incident phrasing. Steps 2 and 4: extract what worked, validate it fresh, then publish.

Knowledge management from chat logs

Chat phrasing. Step 1's monthly mining habit applied to team chat; the third-occurrence rule filters the noise.

Runbook generation from operational history

Runbook phrasing. Same pipeline with runbooks as the output format instead of skills.

How to stop solving the same problem twice

Pain phrasing. The whole skill: the loop from log to skill to link-back is the mechanism.

Why it happens

Operational knowledge is tacit: it lives in the heads of the people who were there and in chat scrollback nobody re-reads. Every repeat incident pays the full learning cost again. Extraction works because the logs already contain the validated procedure; the missing piece was never the knowledge, it was the habit of converting it. The third-occurrence rule focuses effort where the tuition is highest.

Edge cases / pitfalls

• Sanitization is the step that gets skipped under enthusiasm. Never publish a log-derived skill without the privacy pass in step 3; one leaked credential poisons the whole program.
• Some procedures only worked because of undocumented context. The validation run in step 4 exists to catch "worked on that Tuesday" procedures; take its failures seriously.
• Do not mine for blame. If the log review feels like an inquest, people stop writing honest logs; keep the tone curious and the candidates technical.
• Stale skills are worse than missing ones. When the procedure changes, update the skill the same week; a skill that describes last year's incident misleads with authority.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstclYx3oeskE1W-0aYmkXPg