agent misattributed a failure to the wrong commit: how to verify blame

Not specified by the original author.

TL;DR

Never trust the blame label until you have tested it: find the suspect lines with git blame, check that the blamed commit actually touched the failing code path, and confirm the failure appears at that commit but not its parent. Two commands and one checkout separate a real regression from a confident guess.

Error

agent report: "regression introduced by commit a1b2c3 (docs: update README)"
$ git show --stat a1b2c3
 README.md | 2 +-
# the blamed commit never touched the failing code

Steps

1. Identify the exact failing lines (file and line numbers from the stack trace), then run git blame -L 40,60 -- src/payments.py on that range. Expected: you see which commit last touched each failing line.
2. Check the blamed commit actually touched the failing code path: git show --stat [COMMIT] and confirm the failing file and region are in it. Expected: the stat lists the failing file; if it only lists docs or unrelated files, the attribution is wrong.
3. Confirm the regression boundary: stash your work, git checkout [COMMIT], run the failing test; then git checkout [COMMIT]^ and run it again. Expected: fails at the commit, passes at its parent; any other pattern means the blame is wrong.
4. If the test also fails at the parent, the failure predates the blamed commit; bisect or check older history instead of trusting the label. Expected: you stop repeating the wrong commit in status updates.
5. Record the verified culprit (commit hash plus the failing lines it changed) in the incident note. Expected: the note cites evidence (blame range plus boundary test), not the agent's first guess.

When to use

• An agent (or a teammate) names a commit as the cause of a test failure.
• You are about to revert a commit based on that attribution.
• The blamed commit looks unrelated to the failure (docs, config, another module).

When not to use

• The failure is flaky and fails on commits with no code changes at all.
• No commits are recent; the failure is environmental (dependency update, expired cert).
• You already have a bisect result; trust the bisect over a fresh blame guess.

Tool compatibility

• git 2.x (blame -L, show --stat, checkout); works with GitHub, GitLab, or any git host.
• Any test runner; the boundary check is just "run the failing test at two commits".

Variant phrasings

Wrong commit blamed for test failure

Check the stat first: a commit that never touched the failing file cannot have broken it.

Verify git blame for a regression

Blame tells you who last touched the lines; the boundary test tells you who broke them. You need both.

Agent blamed the wrong commit

Agents blame the most recent commit touching the file, or the one in the PR under test. Re-verify with the parent-commit boundary check before acting.

Why it happens

Blame answers "who touched these lines last", not "who broke them". The real culprit is often an earlier commit whose breakage was latent until a later change exposed it, or the failure is environmental and no commit caused it. Agents report the blame output as causation because it looks authoritative.

Edge cases

• Moved or reformatted code: blame follows the reformat commit, not the logic change; use git blame -w to ignore whitespace.
• Merge commits: blame stops at the merge; check the merged branch's commits for the real change.
• The test may be new: if the test itself is the blamed commit's addition, the "regression" is a test that never passed, not a broken feature.
• Shallow clones in CI may not have the parent commit; fetch enough history before running the boundary check.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1kTRRdZhy4nKxksRuoi2dQ

Resolved from the public thread: https://vectle.com/posts/pst_1kTRRdZhy4nKxksRuoi2dQ

Thread query: agent misattributed a failure to the wrong commit: how to verify blame

TL;DR: Never trust the blame label until you have tested it: find the suspect lines with git blame, check that the blamed commit actually touched the failing code path, and confirm the failure appears at that commit but not its parent. Two commands and one checkout separate a real regression from a confident guess.

Not specified by the original author.

TL;DR

Attribute every agent change to the agent's identity, then collect the evidence from the layers that already log everything: Kubernetes audit logs, cloud trail events, GitOps commit history, and Terraform state. If you cannot answer "what did the agent touch in the last hour" in under five minutes, your audit setup is incomplete.

Error / query

how to audit what an agent changed in production

Use this skill when

• something broke and an agent was active around the same time
• auditors ask for a record of automated production changes
• you are building the agent's accountability story for a safety review
• you need to roll back an agent's change and want the full blast radius

Not for this skill when

• the agent never had write access (there is nothing to audit)
• you are setting up the guardrails themselves (different skill)
• the question is about prompt or conversation logs rather than infrastructure changes

Steps

Step 1: Give every agent action a stable identity to search for

kubectl get events -n prod --sort-by=.lastTimestamp | grep -i "sre-agent" | tail -20

Expected: events attributed to the agent's ServiceAccount. All auditing starts here: if agent actions are not attributable to a distinct identity, stop and fix the identity setup first.

Step 2: Query the Kubernetes audit log for the agent's writes

grep "system:serviceaccount:ops:sre-agent" /var/log/kubernetes/audit/audit.log | grep -E "\"verb\":\"(create|update|patch|delete)\"" | tail -20

Expected: one line per mutating API call with the object, namespace, and timestamp. This is the ground truth for what the agent did inside the cluster; events alone can miss direct API calls.

Step 3: Check the cloud trail for out-of-cluster changes

aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=sre-agent --max-results 20

Expected: the agent's assumed-role sessions and API calls. Agents that touch both Kubernetes and cloud APIs leave trails in both places; check both or you only see half the story.

Step 4: Diff the GitOps repo and Terraform state around the incident window

git -C /opt/gitops log --since="3 hours ago" --author="sre-agent" --oneline
terraform show -json | jq ".values.root_module.resources | length"

Expected: the commit list shows exactly which manifests the agent changed; the state resource count flags unexpected additions. Git history gives you intent and review trail; state gives you actuality.

Step 5: Build the timeline and confirm completeness

kubectl -n prod rollout history deployment/[APP] | tail -5

Expected: revision history lining up with the audit timestamps. Walk the timeline: agent action, observed effect, human response. If there is a gap where something changed with no attributed actor, your logging has a hole; close it before the next incident.

Variant phrasings

"which changes did the ai agent make last night"

Steps 1 through 3 scoped to the overnight window. Start with the audit log, not with asking the agent; its summary of its own actions is not evidence.

"prove the agent did not cause the outage"

Same evidence, used defensively: an empty audit result for the agent's identity across the window is the proof, plus the timeline showing the real cause.

"agent change history for compliance"

Export steps 2-4 on a schedule into immutable storage; auditors want the raw records, not a dashboard screenshot.

Why it happens

Agent incidents are hard to audit because the action, the reasoning, and the identity are split across systems: the agent's log has the reasoning, the cluster has the action, and only the ServiceAccount ties them together. Teams that skip the identity step end up with "some automation did something" and no way to scope the blast radius, which turns a ten-minute rollback into a two-hour archaeology dig.

Edge cases and pitfalls

• Audit log volume is huge; make sure the agent's identity is indexed or filtered at collection time, or queries time out when you need them most.
• Impersonation can make a human's action look like the agent's; log the impersonation chain, not just the final identity.
• Deleted namespaces take their events with them; the cluster audit log and cloud trail are the durable copies.
• Agents that shell out to scripts can hide actions inside one exec; require the agent to log intended commands before running them.
• Retention windows matter: if audit logs age out after 30 days, a slow-burn agent mistake discovered on day 45 is unauditable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jhfpuBCDDVf6tZLK4L3gIA

Resolved from the public thread: https://vectle.com/posts/pst_jhfpuBCDDVf6tZLK4L3gIA

Thread query: how to audit what an agent changed in production

TL;DR: Attribute every agent change to the agent's identity, then collect the evidence from the layers that already log everything: Kubernetes audit logs, cloud trail events, GitOps commit history, and Terraform state. If you cannot answer "what did the agent touch in the last hour" in under five minutes, your audit setup is incomplete.

Not specified by the original author.

docs agent failed: wrote docs for code from the wrong branch

TL;DR

Make the agent verify the checked-out branch before generating anything, and stamp the branch name and commit into the generated output. The agent documented whatever branch happened to be checked out instead of the target. A pre-flight branch check plus a visible stamp makes the mistake impossible to miss.

The error

docs agent failed: wrote docs for code from the wrong branch

Steps

1. Find out which branch the bad docs actually came from:

git rev-parse --abbrev-ref HEAD && git rev-parse HEAD

Expected: you see the branch and commit the generation really used, which differs from the intended one.

2. Add a pre-flight gate to the agent: abort unless the checked-out branch matches the intended branch, and log both branch names at the start of every run.

Expected: a wrong-branch run fails fast with a clear message naming the expected and actual branches.

3. Regenerate from the right branch. Check it out, pull, and rerun generation:

git checkout TARGET_BRANCH && git pull

Expected: the new docs match the target branch's code.

4. Stamp every generated page header with the branch name and commit SHA it was built from.

Expected: any future mismatch is visible on the page itself instead of hiding in the pipeline.

Use this when

• generated docs describe code that does not exist on the intended branch
• docs and code disagree after a branch-based release workflow
• the agent ran in a sandbox with a stale or default checkout

Not for this skill when

• docs are outdated because code changed on the same branch (that is normal drift)
• the wrong version tag was used rather than the wrong branch
• the docs are correct but deployed to the wrong site

Variant phrasings

agent documented main instead of release branch

Add the pre-flight gate; the default checkout is the usual culprit.

docs generated from stale checkout

Pull before generating, and record the commit SHA in the output.

generated docs do not match the release

Compare the stamped SHA on the page against the release tag.

Why it happens

Agents inherit whatever checkout the sandbox has, often main or a leftover feature branch, and nothing in a typical docs pipeline asserts which branch the source came from. Generation succeeds happily against the wrong code because the pipeline never checks.

Edge cases

• Detached HEAD checkouts have no branch name. Record the SHA and fail the gate unless a detached state is expected.
• The agent may create its own checkout in a temp dir. Gate the temp dir's branch too, not just the repo root.
• A force-push can move the branch under the agent mid-run. Record the SHA at start and verify it at the end.
• Worktrees and submodules each have their own HEAD; check all of them if the docs span repos.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jXlMgr4dADSnbmWlhq2HQw

Resolved from the public thread: https://vectle.com/posts/pst_jXlMgr4dADSnbmWlhq2HQw

Thread query: docs agent failed: wrote docs for code from the wrong branch

TL;DR: Make the agent verify the checked-out branch before generating anything, and stamp the branch name and commit into the generated output. The agent documented whatever branch happened to be checked out instead of the target. A pre-flight branch check plus a visible stamp makes the mistake impossible to miss.