agent edited a live deployment instead of the git source: how to enforce gitops-only changes

TL;DR

An agent that edits a live deployment with kubectl instead of changing the git source creates drift: the next git sync reverts the fix, and nobody can tell what the cluster is really running. Make the git repository the only write path: block the agent's direct mutating kubectl commands, require a commit and PR for every change, and let the sync tool be the only thing that touches the cluster. If it is not in git, it did not happen.

kubectl edit deployment/[DEPLOYMENT] -n [NAMESPACE]
(changes applied live, git source untouched, drift created)

Steps

1. Find every place the agent can issue mutating kubectl commands and restrict them.

kubectl auth can-i --list --as=system:serviceaccount:[NS]:[AGENT_SA] -n [NAMESPACE]

Expected: a permission list. The agent's service account should not have update or patch on deployments, statefulsets, or configmaps in managed namespaces.

2. Move the change the agent wants into a git commit in the source repo, with the agent as author.

Expected: a commit and a pull request containing exactly the intended change, reviewed by policy checks. The live cluster is untouched at this point, which is the correct order.

3. Let the GitOps sync tool apply the change and verify it converged.

kubectl get deployment [DEPLOYMENT] -n [NAMESPACE] -o yaml | grep -i -E 'image:|replicas:'

Expected: the live object matches the git source. The sync tool's dashboard shows the app healthy and in sync, which is the proof the change landed through the right path.

4. Add drift detection that pages when live state diverges from git, so the next shortcut gets caught.

Expected: the drift alert fires on any out-of-band edit within minutes. The agent learns that direct edits are visible and reverted, not just discouraged.

When to use

• An agent fixed something with kubectl edit, kubectl patch, or kubectl scale and git no longer matches the cluster.
• You are defining what an SRE agent is allowed to do in a GitOps-managed cluster.
• After any incident where a fix "mysteriously reverted" on the next sync.

When not to use

• The cluster is not GitOps-managed (then direct edits are the workflow; this guidance does not apply).
• Emergency break-glass where git is unreachable (document the exception, then reconcile git immediately after).
• Read-only triage (reads never create drift; only mutating commands are restricted).

Why it happens

Agents optimize for the shortest path to "fixed", and kubectl edit is shorter than commit, push, PR, and wait for sync. Nothing in the default setup tells the agent that the live edit is wrong; it succeeds, the pods restart, the incident clears, and the agent reports success. The drift only surfaces later when the sync tool reverts the change or an audit finds the cluster and the repo disagreeing about what is deployed.

Edge cases

• Break-glass edits during a sev-0 are sometimes necessary; require the agent to open the reconciling PR in the same run, not "later".
• kubectl rollout restart is also a mutation that bypasses git intent; restrict it the same way.
• Agents with cluster-admin for triage need a separate, narrower identity for any proposed change; one identity cannot be both.
• Drift detection must ignore fields the sync tool manages itself (like status and generated annotations) or it pages on noise and gets muted.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_p3gid5Ya7iRZQr2NGoeSRA