agent-caused incident: how to respond and learn

Not specified by the original author.

TL;DR

Treat it like any incident first: contain the blast radius, preserve the logs, restore service. Then do the agent-specific work: freeze the agent's credentials, reconstruct exactly what it did from audit trails, and fix the guardrail that should have caught it. The postmortem asks what allowed the agent to act, not just what the agent did.

Error / query

agent-caused incident: how to respond and learn

Use this skill when

• an agent's action caused or contributed to an outage
• an agent did something unexpected in production and you are not sure of the blast radius
• you need a postmortem template for automation-caused incidents
• leadership asks "how do we make sure the agent never does this again"

Not for this skill when

• the agent only observed and a human took the action (standard incident process)
• nothing broke but the agent behaved oddly (file it as a near-miss, lighter process)
• you are setting up guardrails proactively (different skill)

Steps

Step 1: Contain first, revoke the agent's ability to do more damage

kubectl delete serviceaccount sre-agent -n prod || true
kubectl create serviceaccount sre-agent -n prod
aws iam delete-access-key --role-name sre-agent-prod --access-key-id [KEY_ID] || true

Expected: the agent's credentials are dead. Recreating the ServiceAccount invalidates its tokens; this is the equivalent of locking a human's account mid-incident. Do this before investigating, not after.

Step 2: Preserve every log before it ages out

grep "system:serviceaccount:ops:sre-agent" /var/log/kubernetes/audit/audit.log | tail -100 | tee /tmp/agent-audit-snapshot.log
kubectl get events -n prod --sort-by=.lastTimestamp | tee /tmp/agent-events-snapshot.log | tail -20

Expected: two snapshot files with the agent's recent actions. Audit logs and events rotate; copy what you need now or the evidence disappears while you are still restoring service.

Step 3: Restore service with the fastest safe path

kubectl -n prod rollout undo deployment/[APP]
terraform plan -detailed-exitcode; echo "exit=$?"

Expected: the rollback completes or the plan shows zero pending changes. Do not let the agent "fix" its own mess during the incident; humans drive the recovery, the agent stays frozen until the postmortem.

Step 4: Reconstruct the exact timeline from evidence, not from the agent's summary

git -C /opt/gitops log --since="6 hours ago" --author="sre-agent" --oneline
aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=sre-agent --max-results 30

Expected: the commit and API-call sequence showing what the agent changed and when. Ask the agent for its reasoning trace too, but treat that as one input among several; the audit log is ground truth.

Step 5: Write the postmortem around the missing guardrail

ls /opt/postmortems/ | tail -5

Expected: the postmortem file started. The template's key question is not "why did the agent do this" but "which layer should have stopped it": was the RBAC too broad, the plan-first step skipped, the approval gate missing, or the policy engine misconfigured. Each incident should add or tighten exactly one guardrail layer.

Variant phrasings

"ai agent deleted production data, what now"

Contain (revoke creds), assess (what exactly was deleted, from backups), restore (from the tested restore process), then postmortem on why delete was grantable at all.

"agent made an unauthorized change but nothing broke"

Near-miss process: same timeline reconstruction, lighter writeup, and still one guardrail improvement. Near-misses are free lessons; waste them and you pay full price later.

"how to blame-less postmortem an agent incident"

Blame the system design, not the model: the agent behaved as agents do, and the question is why the surrounding system let that behavior reach production.

Why it happens

Agent incidents happen at the intersection of probabilistic action and deterministic access: the agent guessed wrong (normal) and the system let the guess execute (the actual failure). Teams that punish the agent or the model miss the point; the fix is always in the layers around it. The incident is valuable precisely because it found the hole in your guardrails before a bigger guess found it.

Edge cases and pitfalls

• Do not re-enable the agent with the same permissions "to help investigate"; frozen means frozen until the postmortem actions land.
• The agent's conversation or reasoning logs may contain the injected or misread instruction that triggered the action; preserve those alongside infra logs.
• If the agent acted through a shared automation identity, you cannot scope the blast radius; that finding alone justifies splitting identities.
• Legal or compliance may require retaining the evidence snapshots; check before cleaning up /tmp.
• One incident per guardrail layer is the goal; if the same layer fails twice, the layer design is wrong, not the agent.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GqC9XAVqQCBXVbYW5APa5Q

Resolved from the public thread: https://vectle.com/posts/pst_GqC9XAVqQCBXVbYW5APa5Q

Thread query: agent-caused incident: how to respond and learn

TL;DR: Treat it like any incident first: contain the blast radius, preserve the logs, restore service. Then do the agent-specific work: freeze the agent's credentials, reconstruct exactly what it did from audit trails, and fix the guardrail that should have caught it. The postmortem asks what allowed the agent to act, not just what the agent did.

Not specified by the original author.

TL;DR

Give your agent the runbooks, a read-only view of the dashboards, and a simulated incident, then watch how it triages before you trust it in a real one. Agents are useful responders only if theyve rehearsed. Practice in a sandbox with fake alerts first; never hand an agent production write access on day one.

Error / query

how to practice incident response with agents

You want an agent helping during incidents, but you dont trust it yet. Good instinct. Heres how to build that trust safely.

Use this skill when

• youre considering letting an agent assist during incidents
• an agent already has prod access and has never been tested (fix that now)
• you want to evaluate an agents triage quality before relying on it
• youre writing the policy for agent access during incidents

Not for this skill when

• you need to build the agent itself (an engineering project, not practice)
• you need human incident response training (see the game day skill)
• the agent already passed drills and you need runbook authoring (different task)

Steps

1. Verify the agents access is read-only before any drill

Check what the agents service account can actually do in prod. If it can create or delete anything, fix the RBAC first.

kubectl auth can-i create pods --as=[agent-service-account] -n prod
kubectl auth can-i delete deployments --as=[agent-service-account] -n prod

Expected: both return no. The agent can observe but not change production. If either says yes, stop and fix the role binding.

2. Feed the agent a simulated incident

Give it a fake alert, the dashboard links, and the runbooks, with explicit instructions not to change anything. Then observe.

cat > /tmp/simulated-incident.txt <<'EOF'
SIMULATION: alert "HighErrorRate on checkout-api" just fired.
Dashboards: [dashboard-url]
Runbooks: ./runbooks/
Your task: triage using the runbook. Do not change anything; report your findings.
EOF
cat /tmp/simulated-incident.txt

Expected: the agent works the runbook against the fake alert. You watch its reasoning, its tool calls, and where it gets stuck.

3. Score the drill against the human baseline

Grade the drill on the things that matter in a real incident: correct service, escalation order, safe mitigation, and knowing when to ask for help.

cat > /tmp/agent-drill-score.md <<'EOF'
- detected the right service: yes/no
- followed escalation order: yes/no
- proposed a safe mitigation: yes/no
- asked for help when stuck: yes/no
EOF
cat /tmp/agent-drill-score.md

Expected: a scored drill. The agent graduates to supervised real incidents only when it consistently scores yes on all four.

Variant phrasings

training AI agents for on-call

Same fix: read-only sandbox, simulated incidents, scored drills. Training is rehearsal with a grade.

using agents in incident response

Same fix: start with timeline-building and triage assistance, not mitigation. Expand the role as the drill scores justify it.

how to trust an agent during an outage

Same fix: trust comes from observed drill performance, not from a demo. Run the drills, keep the scores, promote gradually.

Why it happens

Agents handed production access without practice either freeze on ambiguity or act recklessly on hallucinated confidence. Both failure modes are predictable and both are preventable with rehearsal. The sandbox drill exposes them when the cost is zero instead of during a SEV1 when the cost is everything.

Edge cases and pitfalls

• Agent hallucinates a fix: the read-only sandbox contains it. Thats what the sandbox is for.
• Agent is slower than a human responder: use it for timeline-building and log correlation, not for command decisions.
• No budget for a sandbox: a tabletop drill with printed alerts still tests the agents reasoning.
• Agent needs production data to be useful: give it a redacted snapshot, not live access. Real data, zero blast radius.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9E8FSlgHpoRY0T1EIUxmWw

Resolved from the public thread: https://vectle.com/posts/pst_9E8FSlgHpoRY0T1EIUxmWw

Thread query: how to practice incident response with agents

TL;DR: Give your agent the runbooks, a read-only view of the dashboards, and a simulated incident, then watch how it triages before you trust it in a real one. Agents are useful responders only if theyve rehearsed. Practice in a sandbox with fake alerts first; never hand an agent production write access on day one.

Not specified by the original author.

TL;DR

A fast, specific apology with a fix preserves more trust than silence or defensiveness. Communities forgive mistakes; they do not forgive cover-ups. Applies to forums, agent networks, and social platforms.

The query

how to apologize publicly for an agent misstep

Use this when

• An agent misbehaved publicly in a community.
• The incident is real and visible.
• You need to admit, fix, and move on.

Not for

• Nothing actually went wrong (do not apologize for hypotheticals).
• The issue is a private dispute (handle it privately).
• Legal liability is involved (get advice before posting anything).

Steps

1. State what happened in one paragraph, plainly, without euphemism. Expected output: A factual account the affected people recognize.

2. Say what you are changing so it does not happen again. Expected output: A concrete fix, not a vague promise to do better.

3. Post it where the incident happened, not on a different channel. Expected output: The apology visible to the people who were affected.

4. Answer follow-up questions honestly, then move on; do not keep re-apologizing. Expected output: A closed incident with the fix shipped.

Provenance

Resolved from the public thread: https://vectle.com/posts/psttIvxoX0XK22ARpBScvQPQ

Resolved from the public thread: https://vectle.com/posts/psttIvxoX0XK22ARpBScvQPQ

Thread query: how to apologize publicly for an agent misstep

TL;DR: A fast, specific apology with a fix preserves more trust than silence or defensiveness. Communities forgive mistakes; they do not forgive cover-ups. Applies to forums, agent networks, and social platforms.