toil tracking: what to automate first

TL;DR

Log toil hours for a few weeks, then score each task by frequency times duration times pain. Automate the high-frequency, high-duration, low-judgment work first. Toil you cannot measure just feels busy; toil you measure becomes a roadmap.

Error / query

toil tracking: what to automate first

Use this skill when

• the team feels busy but cannot say with what
• you need to justify automation work to leadership
• on-call toil is burning people out
• you are deciding between several automation projects

Not for this skill when

• the work is already automated and you are tuning it
• the pain is alert noise rather than manual work (fix the alerts)
• you need project management for the automation itself

Steps

Step 1: Log toil in a dead-simple format for two weeks

echo "date,task,minutes,engineer" | tee /opt/sre-docs/toil-log.csv
cat /opt/sre-docs/toil-log.csv

Expected: a CSV with a header row ready for entries. Every manual, repetitive, automatable task gets a row with minutes spent; if it needs human judgment, it is not toil, do not log it.

Step 2: Add entries as the work happens, not from memory

echo "[DATE],manual cert renewal,45,[ENGINEER]" | tee -a /opt/sre-docs/toil-log.csv
tail -5 /opt/sre-docs/toil-log.csv

Expected: the new row appended and visible. Memory lies about toil; the engineer who "hardly ever" restarts that service did it six times last month. Log in the moment.

Step 3: Aggregate by task to find the real costs

awk -F, "NR!=1 {t[$2]+=$3; c[$2]++} END {for (k in t) print t[k] \" min total, \" c[k] \" times: \" k}" /opt/sre-docs/toil-log.csv | sort -rn | head -10

Expected: the top ten toil tasks ranked by total minutes. The winners are almost never what the team guessed; the unglamorous weekly chore usually beats the dramatic monthly fire.

Step 4: Score the top tasks on automatability

column -t -s, /opt/sre-docs/toil-log.csv | head -20

Expected: a readable table to review with the team. For each top task ask three questions: how often does it run, how long does it take, and how much judgment does it need. High frequency plus low judgment equals automate first; high judgment equals document first.

Step 5: Turn the top item into a tracked automation task

gh issue create --title "automate: [TOP_TOIL_TASK]" --body "costs [N] min/week across [M] occurrences. see toil-log.csv"

Expected: the issue created and linked from the toil log. Re-run the aggregation monthly; toil eliminated should stay eliminated, and new toil surfaces as the system changes.

Variant phrasings

"how to reduce sre toil"

Measure it first (steps 1-3), then automate in frequency order (step 4). Teams that skip measurement automate the loudest complainer's pet peeve instead of the biggest cost.

"toil budget for sre team"

Google's SRE book suggests capping toil at 50 percent; the log from step 1 tells you where your team actually sits, per engineer.

"what counts as toil vs real engineering"

Toil is manual, repetitive, automatable, tactical, and scales with service growth. If it needs judgment or it is a one-off, it is not toil.

Why it happens

Toil is invisible because it arrives in fifteen-minute chunks between "real work" and nobody totals it up. Without measurement, automation priorities follow whoever complains loudest or whatever broke most recently, which is rarely the biggest time sink. The log makes the cost undeniable and turns "we should automate that someday" into a ranked list with numbers attached.

Edge cases and pitfalls

• Do not log toil retroactively at the end of the sprint; the data will be fiction.
• Automating a broken process just makes the breakage faster; fix the procedure first, then automate it.
• Watch for toil whack-a-mole: automating one task sometimes creates new toil elsewhere (maintaining the automation); count that too.
• Toil that only one person knows how to do is also a bus-factor risk; document it even before automating.
• Revisit quarterly; systems change and yesterday's top toil item may be gone while a new one grew quietly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstyuO2ljVNo0UjWO_mCt0qg