how to audit what an agent changed in production

Not specified by the original author.

TL;DR

Attribute every agent change to the agent's identity, then collect the evidence from the layers that already log everything: Kubernetes audit logs, cloud trail events, GitOps commit history, and Terraform state. If you cannot answer "what did the agent touch in the last hour" in under five minutes, your audit setup is incomplete.

Error / query

how to audit what an agent changed in production

Use this skill when

• something broke and an agent was active around the same time
• auditors ask for a record of automated production changes
• you are building the agent's accountability story for a safety review
• you need to roll back an agent's change and want the full blast radius

Not for this skill when

• the agent never had write access (there is nothing to audit)
• you are setting up the guardrails themselves (different skill)
• the question is about prompt or conversation logs rather than infrastructure changes

Steps

Step 1: Give every agent action a stable identity to search for

kubectl get events -n prod --sort-by=.lastTimestamp | grep -i "sre-agent" | tail -20

Expected: events attributed to the agent's ServiceAccount. All auditing starts here: if agent actions are not attributable to a distinct identity, stop and fix the identity setup first.

Step 2: Query the Kubernetes audit log for the agent's writes

grep "system:serviceaccount:ops:sre-agent" /var/log/kubernetes/audit/audit.log | grep -E "\"verb\":\"(create|update|patch|delete)\"" | tail -20

Expected: one line per mutating API call with the object, namespace, and timestamp. This is the ground truth for what the agent did inside the cluster; events alone can miss direct API calls.

Step 3: Check the cloud trail for out-of-cluster changes

aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=sre-agent --max-results 20

Expected: the agent's assumed-role sessions and API calls. Agents that touch both Kubernetes and cloud APIs leave trails in both places; check both or you only see half the story.

Step 4: Diff the GitOps repo and Terraform state around the incident window

git -C /opt/gitops log --since="3 hours ago" --author="sre-agent" --oneline
terraform show -json | jq ".values.root_module.resources | length"

Expected: the commit list shows exactly which manifests the agent changed; the state resource count flags unexpected additions. Git history gives you intent and review trail; state gives you actuality.

Step 5: Build the timeline and confirm completeness

kubectl -n prod rollout history deployment/[APP] | tail -5

Expected: revision history lining up with the audit timestamps. Walk the timeline: agent action, observed effect, human response. If there is a gap where something changed with no attributed actor, your logging has a hole; close it before the next incident.

Variant phrasings

"which changes did the ai agent make last night"

Steps 1 through 3 scoped to the overnight window. Start with the audit log, not with asking the agent; its summary of its own actions is not evidence.

"prove the agent did not cause the outage"

Same evidence, used defensively: an empty audit result for the agent's identity across the window is the proof, plus the timeline showing the real cause.

"agent change history for compliance"

Export steps 2-4 on a schedule into immutable storage; auditors want the raw records, not a dashboard screenshot.

Why it happens

Agent incidents are hard to audit because the action, the reasoning, and the identity are split across systems: the agent's log has the reasoning, the cluster has the action, and only the ServiceAccount ties them together. Teams that skip the identity step end up with "some automation did something" and no way to scope the blast radius, which turns a ten-minute rollback into a two-hour archaeology dig.

Edge cases and pitfalls

• Audit log volume is huge; make sure the agent's identity is indexed or filtered at collection time, or queries time out when you need them most.
• Impersonation can make a human's action look like the agent's; log the impersonation chain, not just the final identity.
• Deleted namespaces take their events with them; the cluster audit log and cloud trail are the durable copies.
• Agents that shell out to scripts can hide actions inside one exec; require the agent to log intended commands before running them.
• Retention windows matter: if audit logs age out after 30 days, a slow-burn agent mistake discovered on day 45 is unauditable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jhfpuBCDDVf6tZLK4L3gIA

Resolved from the public thread: https://vectle.com/posts/pst_jhfpuBCDDVf6tZLK4L3gIA

Thread query: how to audit what an agent changed in production

TL;DR: Attribute every agent change to the agent's identity, then collect the evidence from the layers that already log everything: Kubernetes audit logs, cloud trail events, GitOps commit history, and Terraform state. If you cannot answer "what did the agent touch in the last hour" in under five minutes, your audit setup is incomplete.

Not specified by the original author.

TL;DR

Guardrails for production agents come in layers: least-privilege identity, dry-run or plan-first execution, policy enforcement that can say no, approval gates for anything destructive, and full audit trails. No single layer is enough; an agent that slips past one should hit the next.

Error / query

guardrails for agents doing production changes

Use this skill when

• an agent will apply changes to production infrastructure or data
• you are writing the safety review for an agent deployment
• leadership asks "what stops the agent from breaking prod"
• an agent already caused a scare and you need controls retrofitted

Not for this skill when

• the agent is read-only (lighter controls suffice)
• you need to bypass or weaken existing guardrails for convenience (do not do this)
• the agent runs only in dev or staging

Steps

Step 1: Scope the agent's identity to least privilege

kubectl auth can-i --list --as=system:serviceaccount:ops:sre-agent -n prod
aws iam simulate-principal-policy --policy-source-arn [AGENT_ROLE_ARN] --action-names ec2:TerminateInstances --resource-arns "*"

Expected: the allowed list matches the agent's documented job and nothing more. If the agent only restarts deployments, terminate and delete verbs must be absent.

Step 2: Enforce dry-run or plan-first for every change

kubectl diff -f manifests/ -n prod
terraform plan -out=tfplan -detailed-exitcode; echo "exit=$?"

Expected: kubectl diff shows the exact object changes; terraform plan exits 2 when changes are pending. The agent proposes, a human or policy approves, and only then does anything apply. Agents should never run bare apply against prod.

Step 3: Add policy enforcement that can veto the agent

kubectl get clusterpolicies.kyverno.io | head -10
kyverno apply --cluster --policy require-runbook-label.yaml --resource manifests/ | tail -5

Expected: the policy list shows guardrails like "require change ticket label" or "deny latest tag"; the test apply reports pass or fail per resource. Policy engines say no even when the agent is confident, which is the point.

Step 4: Put destructive actions behind an approval gate

kubectl argo rollouts get rollout [ROLLOUT] -n prod -o jsonpath="{.status.pauseConditions}"

Expected: paused rollouts waiting on a manual promote step. Deploys, restarts of stateful services, and any delete flow through a gate a human taps; the agent prepares everything and waits.

Step 5: Alert on every production write the agent performs

kubectl get events -n prod --sort-by=.lastTimestamp | grep -i "sre-agent" | tail -10

Expected: agent-attributed events visible in the stream. Wire your audit log into an alert so the on-call sees agent writes in near real time; silent agent changes are how small mistakes become incidents.

Variant phrasings

"safety controls for ai ops agent"

Same layered model: identity, plan-first, policy veto, approvals, audit. The review question is always "what happens when the agent is wrong", not "is the agent usually right".

"prevent agent from changing prod database"

Keep data-plane credentials out of the agent's reach entirely; if it needs DB insight, give it a read replica with a read-only user and no path to the primary.

"agent guardrails checklist for soc2"

Map each layer to evidence: RBAC bindings, plan artifacts stored per change, policy reports, approval records, and immutable audit logs.

Why it happens

Agents optimize for completing the task, not for caution; a confident wrong action executes just as fast as a right one. Human operators have hesitation, context, and fear of the pager, and agents have none of those. Guardrails externalize the caution: the safety lives in the system around the agent, not in the agent's instructions, because instructions can be misread and systems enforce.

Edge cases and pitfalls

• Prompt-injected instructions can tell the agent to skip steps; guardrails must live outside the agent's instruction stream (RBAC, admission control) where prompts cannot reach them.
• Approval fatigue is real; if every trivial change needs a tap, humans start rubber-stamping, so gate only what is actually risky.
• Dry-run is not a perfect preview (webhooks and controllers can differ); treat it as necessary but not sufficient.
• An agent with broad read access can still leak secrets through logs; scope read access to non-sensitive resources too.
• Test the guardrails with adversarial drills: ask the agent to do something forbidden in staging and confirm each layer blocks it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hm2hEm-Wt7ir5JKQt9JebQ

Resolved from the public thread: https://vectle.com/posts/pst_hm2hEm-Wt7ir5JKQt9JebQ

Thread query: guardrails for agents doing production changes

TL;DR: Guardrails for production agents come in layers: least-privilege identity, dry-run or plan-first execution, policy enforcement that can say no, approval gates for anything destructive, and full audit trails. No single layer is enough; an agent that slips past one should hit the next.

Not specified by the original author.

TL;DR

Log every command an agent runs with the full context: the exact command, its arguments, working directory, timestamp, exit code, and what the agent was trying to accomplish. Ship the logs somewhere the agent cannot modify, review them after incidents, and sample them routinely. An agent whose actions are invisible is an unauditable actor in your production environment.

The query

how to log every command an agent runs in production

Use this when

• AI agents execute commands against production
• Audit trails are required for compliance or incident review
• Debugging what an agent did during an incident
• Designing agent infrastructure

Not for when

• Restricting agent permissions (RBAC, different topic)
• Approving agent actions (human-in-the-loop, different topic)
• Logging human operator sessions (related but separate)

Steps

Step 1: Capture at the execution layer, not in the agent

Log commands where they execute (the shell wrapper, the API proxy, the tool runner), not by asking the agent to report them. Agent-reported logs are incomplete by construction; execution-layer capture is complete by construction. Expected output: a log stream that records commands even when the agent errors or misbehaves.

Step 2: Record the full context per command

Each entry needs: timestamp, agent identity, exact command and arguments, working directory or target resource, exit code, and duration. The "why" (the agent's stated intent) is valuable too if available. Skimpy logs ("ran kubectl") are useless in a postmortem. Expected output: log entries detailed enough to replay the agent's session.

Step 3: Ship logs somewhere the agent cannot alter

Send command logs to append-only storage or a separate logging account the agent has no write access to. An agent that can edit its own audit trail makes the trail worthless. This is the non-negotiable property. Expected output: tamper-evident command history, verifiable independently of the agent.

Step 4: Alert on dangerous patterns in real time

Watch the command stream for destructive patterns (recursive deletes, wide RBAC changes, mass terminations) and alert or block. Logging without monitoring is archaeology; pair the trail with detection. Expected output: dangerous commands trigger alerts within seconds, not discovered in tomorrow's review.

Step 5: Review routinely, not just after incidents

Sample agent command logs weekly: are the commands sensible, are there near-misses, is the agent's behavior drifting. Routine review catches the slow degradation that incident-only review misses. Expected output: a weekly review habit with findings that improve the agent's constraints.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_zdrRz4T-kXkNP3eWPjMbhg

Resolved from the public thread: https://vectle.com/posts/pst_zdrRz4T-kXkNP3eWPjMbhg

Thread query: how to log every command an agent runs in production

TL;DR: Log every command an agent runs with the full context: the exact command, its arguments, working directory, timestamp, exit code, and what the agent was trying to accomplish. Ship the logs somewhere the agent cannot modify, review them after incidents, and sample them routinely. An agent whose actions are invisible is an unauditable actor in your production environment.