how to measure agent action success rate in incidents

TL;DR

Measure agent actions the way you measure responder actions: what fraction of actions achieved their intent without human correction, how long they took, and what fraction made things worse. Track it per incident and in aggregate, separate routine actions from judgment calls, and let the numbers (not demos) decide how much autonomy the agent earns. Success rate without severity weighting is vanity.

The query

how to measure agent action success rate in incidents

Use this when

• AI agents act during incident response
• Evaluating whether agents help or hurt MTTR
• Deciding autonomy levels for incident agents
• Reporting on agent reliability to leadership

Not for when

• General-purpose agent benchmarks
• Measuring human responder performance
• Non-incident agent task success

Steps

Step 1: Log every agent action with its intent

Record what the agent did and what it was trying to achieve, per action. Without the intent, you cannot judge success: a restarted pod is success if the intent was recovery, failure if the intent was diagnosis. Expected output: an action log with intents, captured during the incident.

Step 2: Score outcomes honestly after the incident

For each action, mark: succeeded unaided, succeeded with human correction, no effect, or made things worse. Score in the postmortem while memories are fresh, with the humans who supervised. Be brutal; inflated scores buy false confidence. Expected output: per-action scores, agreed by the review participants.

Step 3: Weight by blast radius, not just count

One bad database drop outweighs fifty successful log reads. Track success rate separately for read-only, standard-change, and high-blast-radius actions. The aggregate number hides the risk distribution; the segmented numbers reveal it. Expected output: success rates per risk tier, showing where the agent is trustworthy and where it is not.

Step 4: Track trends across incidents

Plot the rates over time: is the agent improving as guardrails and prompts improve, or flat? Correlate changes in success rate with changes you made (better runbooks, tighter scopes). Trends justify continued investment or trigger redesign. Expected output: a trend line with annotations for what changed and when.

Step 5: Tie autonomy to the numbers

Set explicit thresholds: above X% unaided success on standard changes earns broader scope; below Y% on high-blast-radius actions revokes it. Autonomy becomes a function of measured reliability, not of optimism. Expected output: autonomy levels with numeric gates, reviewed as the numbers move.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstBldiuADRoVadpQCsThUqw