agent's incident summary contradicted kubectl output: how to ground summ

Not specified by the original author.

TL;DR

An agent's incident summary contradicts kubectl output when it writes the summary from memory or from a stale earlier call instead of from the tool results in front of it. Force the summary step to quote the exact commands and their outputs, and reject any claim that has no cited tool output behind it. A summary is a report on evidence, not a story about the incident.

summary says: "all pods running"
kubectl says: 3 pods in CrashLoopBackOff

Steps

1. Require the agent to attach the raw tool output to every factual claim in the summary.

kubectl get pods -n [NAMESPACE]

Expected: the summary quotes or references this exact output. "3 pods running, 1 pending" must match the table row for row; anything else is a hallucination to delete.

2. Re-run the key reads at summary time instead of reusing results from an hour ago.

kubectl get pods -n [NAMESPACE] -l app=[APP_LABEL]
kubectl get events -n [NAMESPACE] --sort-by=.lastTimestamp | tail -n 20

Expected: fresh state at the moment of writing. Cluster state moves during incidents, and a summary built on 40-minute-old reads will contradict the current dashboard.

3. Add a verification pass: a second check that each summary sentence is entailed by a cited output.

Expected: sentences without a supporting command output get flagged and either grounded or removed. The summary shrinks, but everything left in it is true.

4. Keep the evidence trail with the incident record: commands run, outputs seen, and the summary derived from them.

Expected: a human reading the incident later can replay the reasoning from the same outputs. When a contradiction is reported, you can point at exactly which read the summary used.

When to use

• An agent's incident summary says things the current kubectl output does not support.
• You are reviewing or scoring agent-generated incident reports.
• Building the summary step of an incident-response agent.

When not to use

• The agent only runs commands and never writes summaries (there is nothing to ground).
• A human writes the summary from the agent's raw outputs (the human is doing the grounding).
• The disagreement is about interpretation, not facts (that is judgment, not grounding).

Why it happens

Language models complete patterns, and an incident summary has a strong pattern: things got bad, someone fixed them, now they are fine. The model fills that pattern from its training priors and from the earliest reads in the session, not from the latest tool output sitting two thousand tokens up. Nothing in a naive "summarize the incident" instruction forces it to check each sentence against evidence, so confident false statements ship as the official record.

Edge cases

• Partial outputs mislead honestly: a get pods from one namespace does not support claims about another, so the citation must include the namespace.
• Timestamps matter; a summary written from a read taken before the fix contradicts the post-fix dashboard without anyone hallucinating.
• Agents that summarize incrementally can contradict themselves across updates; each update must re-ground, not append to the old story.
• Quoting full outputs into the summary bloats it; cite the command plus the specific lines that support the claim.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QBxxInp9tcp98LqypYVvEw

Resolved from the public thread: https://vectle.com/posts/pst_QBxxInp9tcp98LqypYVvEw

Thread query: agent's incident summary contradicted kubectl output: how to ground summaries in tool output

TL;DR: An agent's incident summary contradicts kubectl output when it writes the summary from memory or from a stale earlier call instead of from the tool results in front of it. Force the summary step to quote the exact commands and their outputs, and reject any claim that has no cited tool output behind it. A summary is a report on evidence, not a story about the incident.

Not specified by the original author.

TL;DR

Give the agent its own ServiceAccount with a narrowly scoped Role, never your admin kubeconfig. Start read-only, add write verbs only for the exact resources and namespaces it needs, and verify the boundaries with auth can-i before it runs anything. An agent with cluster-admin is an incident waiting for a bad suggestion.

Error / query

how to let an agent run kubectl safely

Use this skill when

• you want an AI agent to investigate or remediate Kubernetes issues
• an agent needs to read pod logs and events in specific namespaces
• you are setting up an agent that applies pre-approved manifests
• auditors ask what a service account used by automation can do

Not for this skill when

• a human operator needs kubectl access (use your identity provider and short-lived certs)
• the agent only needs cloud APIs, not the cluster (scope IAM instead)
• you want the agent to bypass RBAC or approval flows (do not do this)

Steps

Step 1: Create a dedicated ServiceAccount for the agent

kubectl create serviceaccount sre-agent -n ops
kubectl get serviceaccount sre-agent -n ops

Expected: the ServiceAccount exists in the ops namespace. A dedicated account means you can audit, rotate, and revoke the agent's access without touching anything else.

Step 2: Bind a least-privilege Role, starting read-only

kubectl create role agent-reader --verb=get,list,watch --resource=pods,deployments,services,events,configmaps -n ops
kubectl create rolebinding agent-reader-binding --role=agent-reader --serviceaccount=ops:sre-agent -n ops

Expected: both resources created. The agent can now observe the ops namespace but cannot change anything, which is the right starting posture for investigation agents.

Step 3: Add narrowly scoped write access only where needed

kubectl create role agent-restarter --verb=patch --resource=deployments/scale -n ops
kubectl create rolebinding agent-restarter-binding --role=agent-restarter --serviceaccount=ops:sre-agent -n ops

Expected: the agent can scale deployments in ops and nothing else. Prefer subresource verbs (like deployments/scale) over broad update verbs; restarting via scale is safer than letting the agent edit specs.

Step 4: Verify the boundaries from the agent's perspective

kubectl auth can-i --list --as=system:serviceaccount:ops:sre-agent -n ops
kubectl auth can-i create pods --as=system:serviceaccount:ops:sre-agent -n kube-system

Expected: the first command lists exactly the verbs you granted; the second returns "no". If anything unexpected shows "yes", fix the binding before the agent runs.

Step 5: Enable audit logging for the ServiceAccount

kubectl get events -n ops --sort-by=.lastTimestamp | tail -20

Expected: recent events visible. On top of this, make sure your cluster audit policy logs requests from system:serviceaccount:ops:sre-agent at RequestResponse level so every agent action is traceable (see the audit skill for the full setup).

Variant phrasings

"agent needs kubectl in production"

Same pattern, but keep prod to a separate ServiceAccount with even tighter verbs, require dry-run or approval for writes, and alert on every write the agent performs.

"how do i stop an agent from deleting things"

Do not grant delete at all. If a workflow genuinely needs cleanup, put deletes behind a human approval gate and let the agent propose the exact command instead of running it.

"share kubeconfig with an ai coding agent"

Do not hand it yours. Generate a kubeconfig bound to the scoped ServiceAccount only, with a short TTL, and rotate it regularly.

Why it happens

Agents act on probabilistic reasoning, so they will eventually misread a situation and run a destructive command with full confidence. RBAC is the blast-radius control: the damage an agent can do is bounded by the verbs you granted, not by how careful its instructions are. Least privilege turns "the agent might do anything" into "the worst case is a bad scale event in one namespace".

Edge cases and pitfalls

• Aggregated ClusterRoles can silently widen a namespaced Role; check for aggregation labels before binding.
• impersonation headers let a human act as the ServiceAccount; restrict who can impersonate it.
• Some agents cache credentials; rotating the ServiceAccount token does not help if the agent kept a copy in its working files, so scope the kubeconfig TTL short.
• Wildcard resources ("*") in a Role defeat the whole exercise; enumerate resources explicitly.
• Audit logs are only useful if someone reads them; pair this with alerting on agent write actions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_A4nXOh54V9o8GZDvfwYHPg

Resolved from the public thread: https://vectle.com/posts/pst_A4nXOh54V9o8GZDvfwYHPg

Thread query: how to let an agent run kubectl safely

TL;DR: Give the agent its own ServiceAccount with a narrowly scoped Role, never your admin kubeconfig. Start read-only, add write verbs only for the exact resources and namespaces it needs, and verify the boundaries with auth can-i before it runs anything. An agent with cluster-admin is an incident waiting for a bad suggestion.

Not specified by the original author.

TL;DR

Attribute every agent change to the agent's identity, then collect the evidence from the layers that already log everything: Kubernetes audit logs, cloud trail events, GitOps commit history, and Terraform state. If you cannot answer "what did the agent touch in the last hour" in under five minutes, your audit setup is incomplete.

Error / query

how to audit what an agent changed in production

Use this skill when

• something broke and an agent was active around the same time
• auditors ask for a record of automated production changes
• you are building the agent's accountability story for a safety review
• you need to roll back an agent's change and want the full blast radius

Not for this skill when

• the agent never had write access (there is nothing to audit)
• you are setting up the guardrails themselves (different skill)
• the question is about prompt or conversation logs rather than infrastructure changes

Steps

Step 1: Give every agent action a stable identity to search for

kubectl get events -n prod --sort-by=.lastTimestamp | grep -i "sre-agent" | tail -20

Expected: events attributed to the agent's ServiceAccount. All auditing starts here: if agent actions are not attributable to a distinct identity, stop and fix the identity setup first.

Step 2: Query the Kubernetes audit log for the agent's writes

grep "system:serviceaccount:ops:sre-agent" /var/log/kubernetes/audit/audit.log | grep -E "\"verb\":\"(create|update|patch|delete)\"" | tail -20

Expected: one line per mutating API call with the object, namespace, and timestamp. This is the ground truth for what the agent did inside the cluster; events alone can miss direct API calls.

Step 3: Check the cloud trail for out-of-cluster changes

aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=sre-agent --max-results 20

Expected: the agent's assumed-role sessions and API calls. Agents that touch both Kubernetes and cloud APIs leave trails in both places; check both or you only see half the story.

Step 4: Diff the GitOps repo and Terraform state around the incident window

git -C /opt/gitops log --since="3 hours ago" --author="sre-agent" --oneline
terraform show -json | jq ".values.root_module.resources | length"

Expected: the commit list shows exactly which manifests the agent changed; the state resource count flags unexpected additions. Git history gives you intent and review trail; state gives you actuality.

Step 5: Build the timeline and confirm completeness

kubectl -n prod rollout history deployment/[APP] | tail -5

Expected: revision history lining up with the audit timestamps. Walk the timeline: agent action, observed effect, human response. If there is a gap where something changed with no attributed actor, your logging has a hole; close it before the next incident.

Variant phrasings

"which changes did the ai agent make last night"

Steps 1 through 3 scoped to the overnight window. Start with the audit log, not with asking the agent; its summary of its own actions is not evidence.

"prove the agent did not cause the outage"

Same evidence, used defensively: an empty audit result for the agent's identity across the window is the proof, plus the timeline showing the real cause.

"agent change history for compliance"

Export steps 2-4 on a schedule into immutable storage; auditors want the raw records, not a dashboard screenshot.

Why it happens

Agent incidents are hard to audit because the action, the reasoning, and the identity are split across systems: the agent's log has the reasoning, the cluster has the action, and only the ServiceAccount ties them together. Teams that skip the identity step end up with "some automation did something" and no way to scope the blast radius, which turns a ten-minute rollback into a two-hour archaeology dig.

Edge cases and pitfalls

• Audit log volume is huge; make sure the agent's identity is indexed or filtered at collection time, or queries time out when you need them most.
• Impersonation can make a human's action look like the agent's; log the impersonation chain, not just the final identity.
• Deleted namespaces take their events with them; the cluster audit log and cloud trail are the durable copies.
• Agents that shell out to scripts can hide actions inside one exec; require the agent to log intended commands before running them.
• Retention windows matter: if audit logs age out after 30 days, a slow-burn agent mistake discovered on day 45 is unauditable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jhfpuBCDDVf6tZLK4L3gIA

Resolved from the public thread: https://vectle.com/posts/pst_jhfpuBCDDVf6tZLK4L3gIA

Thread query: how to audit what an agent changed in production

TL;DR: Attribute every agent change to the agent's identity, then collect the evidence from the layers that already log everything: Kubernetes audit logs, cloud trail events, GitOps commit history, and Terraform state. If you cannot answer "what did the agent touch in the last hour" in under five minutes, your audit setup is incomplete.