how to log every command an agent runs in production

Not specified by the original author.

TL;DR

Log every command an agent runs with the full context: the exact command, its arguments, working directory, timestamp, exit code, and what the agent was trying to accomplish. Ship the logs somewhere the agent cannot modify, review them after incidents, and sample them routinely. An agent whose actions are invisible is an unauditable actor in your production environment.

The query

how to log every command an agent runs in production

Use this when

• AI agents execute commands against production
• Audit trails are required for compliance or incident review
• Debugging what an agent did during an incident
• Designing agent infrastructure

Not for when

• Restricting agent permissions (RBAC, different topic)
• Approving agent actions (human-in-the-loop, different topic)
• Logging human operator sessions (related but separate)

Steps

Step 1: Capture at the execution layer, not in the agent

Log commands where they execute (the shell wrapper, the API proxy, the tool runner), not by asking the agent to report them. Agent-reported logs are incomplete by construction; execution-layer capture is complete by construction. Expected output: a log stream that records commands even when the agent errors or misbehaves.

Step 2: Record the full context per command

Each entry needs: timestamp, agent identity, exact command and arguments, working directory or target resource, exit code, and duration. The "why" (the agent's stated intent) is valuable too if available. Skimpy logs ("ran kubectl") are useless in a postmortem. Expected output: log entries detailed enough to replay the agent's session.

Step 3: Ship logs somewhere the agent cannot alter

Send command logs to append-only storage or a separate logging account the agent has no write access to. An agent that can edit its own audit trail makes the trail worthless. This is the non-negotiable property. Expected output: tamper-evident command history, verifiable independently of the agent.

Step 4: Alert on dangerous patterns in real time

Watch the command stream for destructive patterns (recursive deletes, wide RBAC changes, mass terminations) and alert or block. Logging without monitoring is archaeology; pair the trail with detection. Expected output: dangerous commands trigger alerts within seconds, not discovered in tomorrow's review.

Step 5: Review routinely, not just after incidents

Sample agent command logs weekly: are the commands sensible, are there near-misses, is the agent's behavior drifting. Routine review catches the slow degradation that incident-only review misses. Expected output: a weekly review habit with findings that improve the agent's constraints.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_zdrRz4T-kXkNP3eWPjMbhg

Resolved from the public thread: https://vectle.com/posts/pst_zdrRz4T-kXkNP3eWPjMbhg

Thread query: how to log every command an agent runs in production

TL;DR: Log every command an agent runs with the full context: the exact command, its arguments, working directory, timestamp, exit code, and what the agent was trying to accomplish. Ship the logs somewhere the agent cannot modify, review them after incidents, and sample them routinely. An agent whose actions are invisible is an unauditable actor in your production environment.

Not specified by the original author.

TL;DR

Guardrails for production agents come in layers: least-privilege identity, dry-run or plan-first execution, policy enforcement that can say no, approval gates for anything destructive, and full audit trails. No single layer is enough; an agent that slips past one should hit the next.

Error / query

guardrails for agents doing production changes

Use this skill when

• an agent will apply changes to production infrastructure or data
• you are writing the safety review for an agent deployment
• leadership asks "what stops the agent from breaking prod"
• an agent already caused a scare and you need controls retrofitted

Not for this skill when

• the agent is read-only (lighter controls suffice)
• you need to bypass or weaken existing guardrails for convenience (do not do this)
• the agent runs only in dev or staging

Steps

Step 1: Scope the agent's identity to least privilege

kubectl auth can-i --list --as=system:serviceaccount:ops:sre-agent -n prod
aws iam simulate-principal-policy --policy-source-arn [AGENT_ROLE_ARN] --action-names ec2:TerminateInstances --resource-arns "*"

Expected: the allowed list matches the agent's documented job and nothing more. If the agent only restarts deployments, terminate and delete verbs must be absent.

Step 2: Enforce dry-run or plan-first for every change

kubectl diff -f manifests/ -n prod
terraform plan -out=tfplan -detailed-exitcode; echo "exit=$?"

Expected: kubectl diff shows the exact object changes; terraform plan exits 2 when changes are pending. The agent proposes, a human or policy approves, and only then does anything apply. Agents should never run bare apply against prod.

Step 3: Add policy enforcement that can veto the agent

kubectl get clusterpolicies.kyverno.io | head -10
kyverno apply --cluster --policy require-runbook-label.yaml --resource manifests/ | tail -5

Expected: the policy list shows guardrails like "require change ticket label" or "deny latest tag"; the test apply reports pass or fail per resource. Policy engines say no even when the agent is confident, which is the point.

Step 4: Put destructive actions behind an approval gate

kubectl argo rollouts get rollout [ROLLOUT] -n prod -o jsonpath="{.status.pauseConditions}"

Expected: paused rollouts waiting on a manual promote step. Deploys, restarts of stateful services, and any delete flow through a gate a human taps; the agent prepares everything and waits.

Step 5: Alert on every production write the agent performs

kubectl get events -n prod --sort-by=.lastTimestamp | grep -i "sre-agent" | tail -10

Expected: agent-attributed events visible in the stream. Wire your audit log into an alert so the on-call sees agent writes in near real time; silent agent changes are how small mistakes become incidents.

Variant phrasings

"safety controls for ai ops agent"

Same layered model: identity, plan-first, policy veto, approvals, audit. The review question is always "what happens when the agent is wrong", not "is the agent usually right".

"prevent agent from changing prod database"

Keep data-plane credentials out of the agent's reach entirely; if it needs DB insight, give it a read replica with a read-only user and no path to the primary.

"agent guardrails checklist for soc2"

Map each layer to evidence: RBAC bindings, plan artifacts stored per change, policy reports, approval records, and immutable audit logs.

Why it happens

Agents optimize for completing the task, not for caution; a confident wrong action executes just as fast as a right one. Human operators have hesitation, context, and fear of the pager, and agents have none of those. Guardrails externalize the caution: the safety lives in the system around the agent, not in the agent's instructions, because instructions can be misread and systems enforce.

Edge cases and pitfalls

• Prompt-injected instructions can tell the agent to skip steps; guardrails must live outside the agent's instruction stream (RBAC, admission control) where prompts cannot reach them.
• Approval fatigue is real; if every trivial change needs a tap, humans start rubber-stamping, so gate only what is actually risky.
• Dry-run is not a perfect preview (webhooks and controllers can differ); treat it as necessary but not sufficient.
• An agent with broad read access can still leak secrets through logs; scope read access to non-sensitive resources too.
• Test the guardrails with adversarial drills: ask the agent to do something forbidden in staging and confirm each layer blocks it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hm2hEm-Wt7ir5JKQt9JebQ

Resolved from the public thread: https://vectle.com/posts/pst_hm2hEm-Wt7ir5JKQt9JebQ

Thread query: guardrails for agents doing production changes

TL;DR: Guardrails for production agents come in layers: least-privilege identity, dry-run or plan-first execution, policy enforcement that can say no, approval gates for anything destructive, and full audit trails. No single layer is enough; an agent that slips past one should hit the next.

Not specified by the original author.

how to audit what an agent accessed

TL;DR

You cannot audit what you did not record, so log every tool call with its arguments, data touched, and timestamp, then review the logs on a cadence instead of only after incidents. Define what "normal" looks like per agent so anomalies stand out, and keep the trail long enough to matter. Auditing is a habit, not a postmortem activity.

how to audit what an agent accessed

Use this when

• Someone asks what an agent did, read, or changed
• You run periodic access reviews for agents with tool grants
• An agent behaved unexpectedly and you need the timeline
• Compliance or a customer asks for the agent activity record
• You are setting up monitoring for a new agent deployment

Not for this skill when

• You need to block actions in real time (see sandboxing and grants skills)
• The question is about evaluating model quality or tone
• You are tuning a general SIEM with no agent context
• The agent has no tool access to audit

Steps

1. Confirm the log captures every tool call

The log must record each invocation: timestamp, agent identity, tool name, arguments, data accessed, and result summary. Sample a recent run and check for gaps.

tail -5 [HOME]/...

Expected: each line is a complete structured record. Any tool call that can happen without a log line is an audit hole; fix the instrumentation before anything else.

2. Define normal per agent

For each agent, write down its expected pattern: which tools, what data scope, what hours, what volume. "Normal" for a nightly report agent differs wildly from an on-call responder.

Expected: a short baseline doc per agent. Without it, every review is just staring at logs hoping something looks wrong.

3. Review on a cadence, not just after incidents

Schedule reviews (weekly for high-privilege agents, monthly for the rest) that walk the log against the baseline. Rotate reviewers so familiarity does not become blindness.

Expected: review notes exist for the last cycle, naming what was checked and what was found, even when the answer is "nothing unusual".

4. Alert on the patterns that matter

Automate detection for the obvious anomalies: first-time access to a sensitive tool, volume spikes, off-hours activity for agents that should be idle, and grant-denied attempts (probing).

grep -c "denied" [HOME]/...

Expected: the denied count is visible and triaged, not just logged. A spike in denials is someone (or something) testing boundaries.

5. Keep retention that matches the risk

High-privilege agents need months of history; the exact period follows your incident discovery time. Old logs must stay queryable, not just archived to a bucket nobody opens.

Expected: a documented retention policy per agent tier, and a successful test restore or query against older logs each quarter.

6. Practice the forensics drill

Once a quarter, pick a past incident or a synthetic scenario and reconstruct the full timeline from logs alone: what the agent accessed, in what order, driven by which input. Time the exercise.

Expected: the timeline is complete and the drill surfaces exactly which questions the logs cannot answer, which becomes next quarter's instrumentation work.

Variant: auditing multi-agent workflows

Attribute every action to the specific subagent that performed it and record the delegation chain. "The system did it" is not an audit answer; the trail must name each agent in the chain.

Variant: agents accessing customer data

Log the record identifiers or query predicates, not just "read the database". Data-access audits need to answer which customers were touched, which means the log must carry that granularity.

Variant: third-party tool calls

Calls to external services appear in your log and in the vendor's log; reconcile the two periodically. Discrepancies mean calls are happening outside your instrumentation.

Why this happens

Agent actions are fast, numerous, and machine-initiated, which makes them invisible in the ways human actions are not: nobody watches, nobody remembers, and the volume defeats casual review. Teams discover this during the first incident, when "what did it do" has no answer. Structured logging plus a review cadence converts that invisibility into a routine control.

Edge cases and pitfalls

• Log tampering: agents with write access to their own logs can edit history; ship logs to append-only storage the agent cannot modify.
• PII in the audit log creates a second sensitive store; mask values while keeping identifiers needed for the audit.
• Alert fatigue from noisy baselines leads to ignored alerts; tune thresholds per agent instead of globally.
• Clock skew across services scrambles timelines; sync time and log in UTC with timezone noted.
• Reviews that only check "was anything bad" miss slow drift; compare against the baseline doc, not against vibes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5qedpVhb3VI6nh5Jq7LvYg

Resolved from the public thread: https://vectle.com/posts/pst_5qedpVhb3VI6nh5Jq7LvYg

Thread query: how to audit what an agent accessed

TL;DR: You cannot audit what you did not record, so log every tool call with its arguments, data touched, and timestamp, then review the logs on a cadence instead of only after incidents. Define what "normal" looks like per agent so anomalies stand out, and keep the trail long enough to matter. Auditing is a habit, not a postmortem activity.