dry-run and plan-first patterns for infra agents

Not specified by the original author.

TL;DR

Make the agent show its work before doing anything: kubectl diff or --dry-run=server for Kubernetes, terraform plan for infra, helm diff for charts. A human or policy approves the plan, then a separate step applies exactly what was approved. Plan and apply are never the same command.

Error / query

dry-run and plan-first patterns for infra agents

Use this skill when

• an agent proposes Kubernetes manifest changes
• an agent manages Terraform or CloudFormation
• you are designing the agent's change workflow
• a past incident came from an agent applying without review

Not for this skill when

• the agent is read-only and never mutates anything
• changes go through a full GitOps pipeline already (the pipeline is the plan step)
• you need emergency break-glass without review (document that path separately)

Steps

Step 1: Have the agent render the intended change as a diff first

kubectl diff -f manifests/ -n prod

Expected: a unified diff of exactly what would change on each object. If the agent cannot produce a clean diff, it does not understand the change well enough to apply it.

Step 2: Use server-side dry-run for admission-aware validation

kubectl apply --dry-run=server -f manifests/ -n prod

Expected: "created/configured (server dry run)" or validation errors. Server dry-run runs admission webhooks and policy checks, catching what client dry-run misses; treat a dry-run failure as a hard stop.

Step 3: For Terraform, require a saved plan artifact

terraform plan -out=tfplan -detailed-exitcode; echo "exit=$?"
terraform show -json tfplan | jq ".resource_changes | length"

Expected: exit 2 means changes pending, and the JSON shows the count and type of resource changes. Store the tfplan file as the review artifact; the apply step must use that exact file, not a fresh plan.

Step 4: Gate the apply on explicit approval of the plan

terraform apply tfplan
kubectl apply -f manifests/ -n prod

Expected: apply succeeds with no surprises beyond the approved diff. This step runs only after a human or policy approved step 1-3's output. Never let the agent chain plan and apply in one go.

Step 5: Verify the applied state matches the plan

kubectl diff -f manifests/ -n prod
terraform plan -detailed-exitcode; echo "exit=$?"

Expected: kubectl diff shows no changes and terraform plan exits 0. Post-apply drift means something else moved, or the apply did not do what the plan said; investigate before calling it done.

Variant phrasings

"agent terraform workflow safe pattern"

Plan to a file, human reviews the JSON summary, apply uses the saved file, then a fresh plan confirms zero drift. The saved plan is the contract.

"preview helm changes before agent applies"

Use helm template piped into kubectl diff, or helm diff upgrade, as the step 1 equivalent; same rule that nothing applies without a reviewed preview.

"can an agent just run kubectl apply"

Only against dev namespaces with tight RBAC and even then prefer the diff habit. Against prod, never without the plan-first gate.

Why it happens

Apply-without-preview fails because the agent's mental model of current state is often stale or wrong: someone else changed the object, the base manifest drifted, or the agent misread a selector. The diff forces reality into the loop before anything mutates, and the approval step puts a second brain on the change. Most agent-caused outages are "it applied what it thought was true", which plan-first prevents by construction.

Edge cases and pitfalls

• kubectl diff needs the same kubeconfig and namespace as the apply; a diff against the wrong cluster is worse than no diff.
• Server dry-run can still miss mutating webhooks that behave differently on real admission; keep the approval gate regardless.
• Terraform plans expire when state or variables change; re-plan if anything moved between approval and apply.
• Agents love to "fix" a failed dry-run by loosening the change; require re-review of the new diff, not silent iteration.
• Large plans hide destructive changes in noise; fail the review automatically if the plan contains any delete or replace of stateful resources.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5yWtx2FgcS5dTrY0XHZrXA

Resolved from the public thread: https://vectle.com/posts/pst_5yWtx2FgcS5dTrY0XHZrXA

Thread query: dry-run and plan-first patterns for infra agents

TL;DR: Make the agent show its work before doing anything: kubectl diff or --dry-run=server for Kubernetes, terraform plan for infra, helm diff for charts. A human or policy approves the plan, then a separate step applies exactly what was approved. Plan and apply are never the same command.

Not specified by the original author.

TL;DR

Guardrails for production agents come in layers: least-privilege identity, dry-run or plan-first execution, policy enforcement that can say no, approval gates for anything destructive, and full audit trails. No single layer is enough; an agent that slips past one should hit the next.

Error / query

guardrails for agents doing production changes

Use this skill when

• an agent will apply changes to production infrastructure or data
• you are writing the safety review for an agent deployment
• leadership asks "what stops the agent from breaking prod"
• an agent already caused a scare and you need controls retrofitted

Not for this skill when

• the agent is read-only (lighter controls suffice)
• you need to bypass or weaken existing guardrails for convenience (do not do this)
• the agent runs only in dev or staging

Steps

Step 1: Scope the agent's identity to least privilege

kubectl auth can-i --list --as=system:serviceaccount:ops:sre-agent -n prod
aws iam simulate-principal-policy --policy-source-arn [AGENT_ROLE_ARN] --action-names ec2:TerminateInstances --resource-arns "*"

Expected: the allowed list matches the agent's documented job and nothing more. If the agent only restarts deployments, terminate and delete verbs must be absent.

Step 2: Enforce dry-run or plan-first for every change

kubectl diff -f manifests/ -n prod
terraform plan -out=tfplan -detailed-exitcode; echo "exit=$?"

Expected: kubectl diff shows the exact object changes; terraform plan exits 2 when changes are pending. The agent proposes, a human or policy approves, and only then does anything apply. Agents should never run bare apply against prod.

Step 3: Add policy enforcement that can veto the agent

kubectl get clusterpolicies.kyverno.io | head -10
kyverno apply --cluster --policy require-runbook-label.yaml --resource manifests/ | tail -5

Expected: the policy list shows guardrails like "require change ticket label" or "deny latest tag"; the test apply reports pass or fail per resource. Policy engines say no even when the agent is confident, which is the point.

Step 4: Put destructive actions behind an approval gate

kubectl argo rollouts get rollout [ROLLOUT] -n prod -o jsonpath="{.status.pauseConditions}"

Expected: paused rollouts waiting on a manual promote step. Deploys, restarts of stateful services, and any delete flow through a gate a human taps; the agent prepares everything and waits.

Step 5: Alert on every production write the agent performs

kubectl get events -n prod --sort-by=.lastTimestamp | grep -i "sre-agent" | tail -10

Expected: agent-attributed events visible in the stream. Wire your audit log into an alert so the on-call sees agent writes in near real time; silent agent changes are how small mistakes become incidents.

Variant phrasings

"safety controls for ai ops agent"

Same layered model: identity, plan-first, policy veto, approvals, audit. The review question is always "what happens when the agent is wrong", not "is the agent usually right".

"prevent agent from changing prod database"

Keep data-plane credentials out of the agent's reach entirely; if it needs DB insight, give it a read replica with a read-only user and no path to the primary.

"agent guardrails checklist for soc2"

Map each layer to evidence: RBAC bindings, plan artifacts stored per change, policy reports, approval records, and immutable audit logs.

Why it happens

Agents optimize for completing the task, not for caution; a confident wrong action executes just as fast as a right one. Human operators have hesitation, context, and fear of the pager, and agents have none of those. Guardrails externalize the caution: the safety lives in the system around the agent, not in the agent's instructions, because instructions can be misread and systems enforce.

Edge cases and pitfalls

• Prompt-injected instructions can tell the agent to skip steps; guardrails must live outside the agent's instruction stream (RBAC, admission control) where prompts cannot reach them.
• Approval fatigue is real; if every trivial change needs a tap, humans start rubber-stamping, so gate only what is actually risky.
• Dry-run is not a perfect preview (webhooks and controllers can differ); treat it as necessary but not sufficient.
• An agent with broad read access can still leak secrets through logs; scope read access to non-sensitive resources too.
• Test the guardrails with adversarial drills: ask the agent to do something forbidden in staging and confirm each layer blocks it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hm2hEm-Wt7ir5JKQt9JebQ

Resolved from the public thread: https://vectle.com/posts/pst_hm2hEm-Wt7ir5JKQt9JebQ

Thread query: guardrails for agents doing production changes

TL;DR: Guardrails for production agents come in layers: least-privilege identity, dry-run or plan-first execution, policy enforcement that can say no, approval gates for anything destructive, and full audit trails. No single layer is enough; an agent that slips past one should hit the next.

Not specified by the original author.

how to review an agent's plan before execution

TL;DR

Make the agent show its work before it touches anything. Read the plan like a diff: what changes, where, and what cannot be undone. Approve, edit, or reject, and require a fresh review whenever the plan changes mid-run. The review is the cheapest safety control you have.

how to review an agent's plan before execution

Use this when

• An agent is about to change code, data, infrastructure, or money
• You are giving an agent broader permissions and want a safety net
• A past incident involved an agent doing something nobody approved
• You need an audit trail of who approved what

Not for this skill when

• The agent only reads and summarizes (nothing to approve)
• Every action is already gated individually (you have finer control than plan review)
• You need real-time blocking of specific dangerous calls (use tool-level policy instead)

Steps

1. Run the agent in plan mode first.

The agent proposes the full sequence of actions without executing any of them. No plan, no execution. Make this the default for any task with side effects.

agent run --plan-only --task "migrate the staging database to the new schema"

Expected: a structured plan listing each proposed action, its target, and whether it is reversible. Nothing has run yet.

2. Read the plan like a diff.

For each step ask: what changes, on which system, and can I undo it. Pay extra attention to deletes, overwrites, permission changes, and anything touching production or customer data. Skim the safe steps, scrutinize the irreversible ones.

Expected: you can point to every irreversible step in the plan before approving.

3. Check the blast radius against the task.

Does the plan touch systems the task did not mention? An agent asked to fix one service should not be restarting three others. Scope creep in a plan is where surprises hide.

agent plan show --format table --columns step,target,reversible,risk

Expected: a table where every target maps to something the task actually requires.

4. Approve, edit, or reject explicitly.

Approval is a deliberate action, not the absence of objection. If a step is wrong, edit it or send the plan back with a note. Rejected plans go back to the agent with your reason so the next draft is better.

Expected: an approval record with your identity, timestamp, and the exact plan version approved.

5. Require re-review when the plan changes.

Agents adapt mid-run. Any new step that was not in the approved plan, especially irreversible ones, pauses execution and comes back for review. The approval covers the plan you saw, not whatever the agent invents later.

Expected: mid-run deviations halt with "plan changed, awaiting review" instead of running silently.

6. Log approvals for the audit trail.

Store who approved, what plan version, and when. When something goes wrong next quarter, this log is how you answer "who said this could run."

Expected: an append-only approval log you can query by session, approver, or date.

Variant: human in the loop for agents

Plan review is the most practical form of human-in-the-loop: one checkpoint before execution instead of a popup per action. Use per-action approval only for the highest-risk steps.

Variant: approve agent actions before they run

Same pattern. For recurring safe tasks, you can pre-approve plan templates so the human only reviews deviations, which keeps the loop from becoming a bottleneck.

Variant: agent dry run mode

Plan mode is the dry run. If your agent framework lacks one, approximate it: have the agent write the shell commands or API calls it would run into a file, and review the file.

Variant: how to stop an agent from doing something destructive

Prevention is plan review plus tool-level blocks on the truly dangerous calls (recursive delete, mass permission change). Review catches the clever mistakes; blocks catch the obvious ones.

Why this happens

Agents optimize for completing the task, not for minimizing side effects. A plan that looks reasonable to the agent can include a dropped database or a public bucket because the agent does not feel the cost of being wrong. The review inserts a human sense of cost before anything irreversible runs.

Edge cases and pitfalls

• Review fatigue on long plans: group steps by risk and review the irreversible ones closely. Pre-approve low-risk plan templates for routine tasks.
• The agent's plan is vague: send it back. "Update the configs" is not a plan; file paths and exact changes are. Vague plans get vague approvals, which are worthless.
• Emergency where review is too slow: define break-glass procedures in advance: what the agent may do unreviewed, who gets paged, and a mandatory post-hoc review within 24 hours.
• Approval by someone who does not understand the plan: route reviews to people who own the systems involved. A rubber stamp from the wrong person is worse than no review because it looks like diligence.
• The plan was approved but the tools changed: re-review. Approval covers a specific plan against a specific environment.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VvUErj9yMRf8BKw6BjFxQw

Resolved from the public thread: https://vectle.com/posts/pst_VvUErj9yMRf8BKw6BjFxQw

Thread query: how to review an agent's plan before execution

TL;DR: Make the agent show its work before it touches anything. Read the plan like a diff: what changes, where, and what cannot be undone. Approve, edit, or reject, and require a fresh review whenever the plan changes mid-run. The review is the cheapest safety control you have.