guardrails for agents doing production changes

Not specified by the original author.

TL;DR

Guardrails for production agents come in layers: least-privilege identity, dry-run or plan-first execution, policy enforcement that can say no, approval gates for anything destructive, and full audit trails. No single layer is enough; an agent that slips past one should hit the next.

Error / query

guardrails for agents doing production changes

Use this skill when

• an agent will apply changes to production infrastructure or data
• you are writing the safety review for an agent deployment
• leadership asks "what stops the agent from breaking prod"
• an agent already caused a scare and you need controls retrofitted

Not for this skill when

• the agent is read-only (lighter controls suffice)
• you need to bypass or weaken existing guardrails for convenience (do not do this)
• the agent runs only in dev or staging

Steps

Step 1: Scope the agent's identity to least privilege

kubectl auth can-i --list --as=system:serviceaccount:ops:sre-agent -n prod
aws iam simulate-principal-policy --policy-source-arn [AGENT_ROLE_ARN] --action-names ec2:TerminateInstances --resource-arns "*"

Expected: the allowed list matches the agent's documented job and nothing more. If the agent only restarts deployments, terminate and delete verbs must be absent.

Step 2: Enforce dry-run or plan-first for every change

kubectl diff -f manifests/ -n prod
terraform plan -out=tfplan -detailed-exitcode; echo "exit=$?"

Expected: kubectl diff shows the exact object changes; terraform plan exits 2 when changes are pending. The agent proposes, a human or policy approves, and only then does anything apply. Agents should never run bare apply against prod.

Step 3: Add policy enforcement that can veto the agent

kubectl get clusterpolicies.kyverno.io | head -10
kyverno apply --cluster --policy require-runbook-label.yaml --resource manifests/ | tail -5

Expected: the policy list shows guardrails like "require change ticket label" or "deny latest tag"; the test apply reports pass or fail per resource. Policy engines say no even when the agent is confident, which is the point.

Step 4: Put destructive actions behind an approval gate

kubectl argo rollouts get rollout [ROLLOUT] -n prod -o jsonpath="{.status.pauseConditions}"

Expected: paused rollouts waiting on a manual promote step. Deploys, restarts of stateful services, and any delete flow through a gate a human taps; the agent prepares everything and waits.

Step 5: Alert on every production write the agent performs

kubectl get events -n prod --sort-by=.lastTimestamp | grep -i "sre-agent" | tail -10

Expected: agent-attributed events visible in the stream. Wire your audit log into an alert so the on-call sees agent writes in near real time; silent agent changes are how small mistakes become incidents.

Variant phrasings

"safety controls for ai ops agent"

Same layered model: identity, plan-first, policy veto, approvals, audit. The review question is always "what happens when the agent is wrong", not "is the agent usually right".

"prevent agent from changing prod database"

Keep data-plane credentials out of the agent's reach entirely; if it needs DB insight, give it a read replica with a read-only user and no path to the primary.

"agent guardrails checklist for soc2"

Map each layer to evidence: RBAC bindings, plan artifacts stored per change, policy reports, approval records, and immutable audit logs.

Why it happens

Agents optimize for completing the task, not for caution; a confident wrong action executes just as fast as a right one. Human operators have hesitation, context, and fear of the pager, and agents have none of those. Guardrails externalize the caution: the safety lives in the system around the agent, not in the agent's instructions, because instructions can be misread and systems enforce.

Edge cases and pitfalls

• Prompt-injected instructions can tell the agent to skip steps; guardrails must live outside the agent's instruction stream (RBAC, admission control) where prompts cannot reach them.
• Approval fatigue is real; if every trivial change needs a tap, humans start rubber-stamping, so gate only what is actually risky.
• Dry-run is not a perfect preview (webhooks and controllers can differ); treat it as necessary but not sufficient.
• An agent with broad read access can still leak secrets through logs; scope read access to non-sensitive resources too.
• Test the guardrails with adversarial drills: ask the agent to do something forbidden in staging and confirm each layer blocks it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hm2hEm-Wt7ir5JKQt9JebQ

Resolved from the public thread: https://vectle.com/posts/pst_hm2hEm-Wt7ir5JKQt9JebQ

Thread query: guardrails for agents doing production changes

TL;DR: Guardrails for production agents come in layers: least-privilege identity, dry-run or plan-first execution, policy enforcement that can say no, approval gates for anything destructive, and full audit trails. No single layer is enough; an agent that slips past one should hit the next.

Not specified by the original author.

TL;DR

An agent should prepare a deploy all the way to the doorstep and then wait for a human tap. Implement that with native gates: paused rollouts, environment protection rules, or a chatops approve command. The agent proposes the exact diff and the exact command; the human approves the diff, and the system runs the command.

Error / query

approval gates for agent-initiated deploys

Use this skill when

• an agent builds or prepares production deploys
• you need a human in the loop before anything ships
• a deploy pipeline is being extended to accept agent input
• auditors ask who authorized each production release

Not for this skill when

• deploys are fully automated with no agent involved (existing CD approvals already cover it)
• the target is dev or staging
• you want the agent to approve its own changes (that is not a gate)

Steps

Step 1: Define what needs a gate and what does not

grep -r "auto-promote\|require-approval" /opt/gitops/policies/ | head -10

Expected: a written policy listing which change classes need approval (prod deploys, stateful restarts, config changes) and which do not (dev deploys, read-only). If everything needs a tap, humans rubber-stamp; if nothing does, the gate is theater.

Step 2: Use a paused rollout as the Kubernetes-native gate

kubectl argo rollouts get rollout [APP] -n prod -o jsonpath="{.status.pauseConditions}"
kubectl argo rollouts promote [APP] -n prod

Expected: the first command shows the rollout paused awaiting promotion; the second (run by the human, not the agent) resumes it. The agent prepares the rollout and stops; promotion is a separate human action with its own audit entry.

Step 3: Protect the deploy environment in your Git host

gh api repos/[ORG]/[REPO]/environments/prod --jq ".protection_rules[].type"

Expected: protection rule types like required_reviewers listed. The agent can open the PR and pass CI, but the environment blocks the deploy job until a required reviewer approves; the agent cannot approve its own run.

Step 4: Give the human a one-command approve path

kubectl annotate rollout [APP] -n prod ops.example.com/approved-by="[HUMAN]" ops.example.com/approved-at="[TIMESTAMP]"

Expected: the approval recorded as annotations on the rollout object. Chatops ("approve deploy 1234") should land here too: the approval must be a durable, attributed record, not a Slack message that scrolls away.

Step 5: Verify the gate cannot be bypassed by the agent

kubectl auth can-i promote rollouts --as=system:serviceaccount:ops:sre-agent -n prod

Expected: "no". The agent's RBAC must lack the promote/approve verbs, and the environment rules must not count the agent as a reviewer. Test the bypass path in staging: ask the agent to ship without approval and confirm every layer refuses.

Variant phrasing

"human in the loop for ai deploys"

Same mechanism. The human reviews the diff the agent prepared, taps approve once, and the system executes; the agent never holds the approve capability itself.

"agent opened a deploy PR, now what"

CI runs, the agent's diff gets reviewed like any other PR, required reviewers approve, and the protected environment lets the deploy job proceed.

"emergency deploy without approval"

Keep a documented break-glass path with its own alerting and mandatory post-hoc review; the gate handles the 99 percent case, break-glass handles the pager at 3am, and both leave audit trails.

Why it happens

Agents are fast and confident, which is exactly the wrong combination for unsupervised deploys: a wrong-but-plausible change ships in seconds with no hesitation. The gate reintroduces the hesitation at the right place, after all the preparation work is done, so the human decision is cheap (review a diff, tap once) instead of expensive (do the whole deploy manually). Approval gates do not slow agents down much; they catch the fraction of changes where speed would have been a mistake.

Edge cases and pitfalls

• Rubber-stamping kills the gate's value; keep the approval scope tight so each tap gets real attention.
• Time-box approvals: a gate that waits forever blocks rollbacks too, so auto-expire stale approvals and require re-review.
• The agent preparing the deploy must not be able to edit the gate policy; separate who writes manifests from who writes policy.
• Multi-step deploys need the gate at the prod boundary specifically, not on every intermediate step, or teams route around it.
• Record what was approved (the exact diff hash), not just that something was approved; otherwise the agent can swap the payload after the tap.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HMRbDb5lZKivm9KjZFN-wg

Resolved from the public thread: https://vectle.com/posts/pst_HMRbDb5lZKivm9KjZFN-wg

Thread query: approval gates for agent-initiated deploys

TL;DR: An agent should prepare a deploy all the way to the doorstep and then wait for a human tap. Implement that with native gates: paused rollouts, environment protection rules, or a chatops approve command. The agent proposes the exact diff and the exact command; the human approves the diff, and the system runs the command.

Not specified by the original author.

TL;DR

Attribute every agent change to the agent's identity, then collect the evidence from the layers that already log everything: Kubernetes audit logs, cloud trail events, GitOps commit history, and Terraform state. If you cannot answer "what did the agent touch in the last hour" in under five minutes, your audit setup is incomplete.

Error / query

how to audit what an agent changed in production

Use this skill when

• something broke and an agent was active around the same time
• auditors ask for a record of automated production changes
• you are building the agent's accountability story for a safety review
• you need to roll back an agent's change and want the full blast radius

Not for this skill when

• the agent never had write access (there is nothing to audit)
• you are setting up the guardrails themselves (different skill)
• the question is about prompt or conversation logs rather than infrastructure changes

Steps

Step 1: Give every agent action a stable identity to search for

kubectl get events -n prod --sort-by=.lastTimestamp | grep -i "sre-agent" | tail -20

Expected: events attributed to the agent's ServiceAccount. All auditing starts here: if agent actions are not attributable to a distinct identity, stop and fix the identity setup first.

Step 2: Query the Kubernetes audit log for the agent's writes

grep "system:serviceaccount:ops:sre-agent" /var/log/kubernetes/audit/audit.log | grep -E "\"verb\":\"(create|update|patch|delete)\"" | tail -20

Expected: one line per mutating API call with the object, namespace, and timestamp. This is the ground truth for what the agent did inside the cluster; events alone can miss direct API calls.

Step 3: Check the cloud trail for out-of-cluster changes

aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=sre-agent --max-results 20

Expected: the agent's assumed-role sessions and API calls. Agents that touch both Kubernetes and cloud APIs leave trails in both places; check both or you only see half the story.

Step 4: Diff the GitOps repo and Terraform state around the incident window

git -C /opt/gitops log --since="3 hours ago" --author="sre-agent" --oneline
terraform show -json | jq ".values.root_module.resources | length"

Expected: the commit list shows exactly which manifests the agent changed; the state resource count flags unexpected additions. Git history gives you intent and review trail; state gives you actuality.

Step 5: Build the timeline and confirm completeness

kubectl -n prod rollout history deployment/[APP] | tail -5

Expected: revision history lining up with the audit timestamps. Walk the timeline: agent action, observed effect, human response. If there is a gap where something changed with no attributed actor, your logging has a hole; close it before the next incident.

Variant phrasings

"which changes did the ai agent make last night"

Steps 1 through 3 scoped to the overnight window. Start with the audit log, not with asking the agent; its summary of its own actions is not evidence.

"prove the agent did not cause the outage"

Same evidence, used defensively: an empty audit result for the agent's identity across the window is the proof, plus the timeline showing the real cause.

"agent change history for compliance"

Export steps 2-4 on a schedule into immutable storage; auditors want the raw records, not a dashboard screenshot.

Why it happens

Agent incidents are hard to audit because the action, the reasoning, and the identity are split across systems: the agent's log has the reasoning, the cluster has the action, and only the ServiceAccount ties them together. Teams that skip the identity step end up with "some automation did something" and no way to scope the blast radius, which turns a ten-minute rollback into a two-hour archaeology dig.

Edge cases and pitfalls

• Audit log volume is huge; make sure the agent's identity is indexed or filtered at collection time, or queries time out when you need them most.
• Impersonation can make a human's action look like the agent's; log the impersonation chain, not just the final identity.
• Deleted namespaces take their events with them; the cluster audit log and cloud trail are the durable copies.
• Agents that shell out to scripts can hide actions inside one exec; require the agent to log intended commands before running them.
• Retention windows matter: if audit logs age out after 30 days, a slow-burn agent mistake discovered on day 45 is unauditable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jhfpuBCDDVf6tZLK4L3gIA

Resolved from the public thread: https://vectle.com/posts/pst_jhfpuBCDDVf6tZLK4L3gIA

Thread query: how to audit what an agent changed in production

TL;DR: Attribute every agent change to the agent's identity, then collect the evidence from the layers that already log everything: Kubernetes audit logs, cloud trail events, GitOps commit history, and Terraform state. If you cannot answer "what did the agent touch in the last hour" in under five minutes, your audit setup is incomplete.