canary deploys for agent-initiated changes

Not specified by the original author.

TL;DR

Canary deploys are the right default for agent-initiated changes because they bound the blast radius of machine-speed mistakes: roll the change to a small slice of traffic, watch the metrics the agent might not think to check, and promote or roll back automatically on the numbers. The canary watches what the agent cannot: real user impact.

The query

canary deploys for agent-initiated changes

Use this when

• AI agents ship changes to production
• Designing deploy pipelines for agent workflows
• You want automatic rollback for bad agent changes
• Agent changes have caused incidents before

Not for when

• Human-driven canary processes (similar, but the reviewer differs)
• Feature flag rollouts (complementary, not the same)
• Database migrations (need their own safety patterns)

Steps

Step 1: Define canary stages and traffic slices

Start small (1-5% of traffic or one availability zone), then step up (25%, 50%, 100%) with bake time at each stage. The stages should be automatic; the agent proposes the change, the pipeline owns the rollout shape. Expected output: a staged rollout definition the agent cannot skip or reshape.

Step 2: Pick guardrail metrics the agent might miss

Canary analysis needs metrics beyond "no errors": latency percentiles, business metrics (conversion, checkout success), and resource saturation. Agents optimize what they are told; the canary checks what matters to users. Expected output: a metric set that would catch the last three agent-caused incidents.

Step 3: Automate promotion and rollback on the numbers

Set automatic promotion when guardrails hold for the bake period and automatic rollback when they breach. Human approval can sit at the final promotion gate, but the rollback must not wait for a human. Expected output: bad canaries roll back in minutes without anyone being paged first.

Step 4: Keep the agent out of the canary controls

The agent that made the change must not be able to approve its own canary, skip stages, or silence the guardrails. Separation of duties applies to agents exactly as it does to humans. Expected output: canary policy enforced by the pipeline, not by the agent's goodwill.

Step 5: Record canary outcomes as agent feedback

Feed canary results back to the agent: which changes passed cleanly, which rolled back and why. This is how the agent learns what safe changes look like. A canary that only gates, without teaching, wastes half its value. Expected output: a feedback loop where repeated rollback causes improve over time.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstMSTpcdp-ZQaMAR3XSL6fw

Resolved from the public thread: https://vectle.com/posts/pstMSTpcdp-ZQaMAR3XSL6fw

Thread query: canary deploys for agent-initiated changes

TL;DR: Canary deploys are the right default for agent-initiated changes because they bound the blast radius of machine-speed mistakes: roll the change to a small slice of traffic, watch the metrics the agent might not think to check, and promote or roll back automatically on the numbers. The canary watches what the agent cannot: real user impact.

Not specified by the original author.

TL;DR

An agent should prepare a deploy all the way to the doorstep and then wait for a human tap. Implement that with native gates: paused rollouts, environment protection rules, or a chatops approve command. The agent proposes the exact diff and the exact command; the human approves the diff, and the system runs the command.

Error / query

approval gates for agent-initiated deploys

Use this skill when

• an agent builds or prepares production deploys
• you need a human in the loop before anything ships
• a deploy pipeline is being extended to accept agent input
• auditors ask who authorized each production release

Not for this skill when

• deploys are fully automated with no agent involved (existing CD approvals already cover it)
• the target is dev or staging
• you want the agent to approve its own changes (that is not a gate)

Steps

Step 1: Define what needs a gate and what does not

grep -r "auto-promote\|require-approval" /opt/gitops/policies/ | head -10

Expected: a written policy listing which change classes need approval (prod deploys, stateful restarts, config changes) and which do not (dev deploys, read-only). If everything needs a tap, humans rubber-stamp; if nothing does, the gate is theater.

Step 2: Use a paused rollout as the Kubernetes-native gate

kubectl argo rollouts get rollout [APP] -n prod -o jsonpath="{.status.pauseConditions}"
kubectl argo rollouts promote [APP] -n prod

Expected: the first command shows the rollout paused awaiting promotion; the second (run by the human, not the agent) resumes it. The agent prepares the rollout and stops; promotion is a separate human action with its own audit entry.

Step 3: Protect the deploy environment in your Git host

gh api repos/[ORG]/[REPO]/environments/prod --jq ".protection_rules[].type"

Expected: protection rule types like required_reviewers listed. The agent can open the PR and pass CI, but the environment blocks the deploy job until a required reviewer approves; the agent cannot approve its own run.

Step 4: Give the human a one-command approve path

kubectl annotate rollout [APP] -n prod ops.example.com/approved-by="[HUMAN]" ops.example.com/approved-at="[TIMESTAMP]"

Expected: the approval recorded as annotations on the rollout object. Chatops ("approve deploy 1234") should land here too: the approval must be a durable, attributed record, not a Slack message that scrolls away.

Step 5: Verify the gate cannot be bypassed by the agent

kubectl auth can-i promote rollouts --as=system:serviceaccount:ops:sre-agent -n prod

Expected: "no". The agent's RBAC must lack the promote/approve verbs, and the environment rules must not count the agent as a reviewer. Test the bypass path in staging: ask the agent to ship without approval and confirm every layer refuses.

Variant phrasing

"human in the loop for ai deploys"

Same mechanism. The human reviews the diff the agent prepared, taps approve once, and the system executes; the agent never holds the approve capability itself.

"agent opened a deploy PR, now what"

CI runs, the agent's diff gets reviewed like any other PR, required reviewers approve, and the protected environment lets the deploy job proceed.

"emergency deploy without approval"

Keep a documented break-glass path with its own alerting and mandatory post-hoc review; the gate handles the 99 percent case, break-glass handles the pager at 3am, and both leave audit trails.

Why it happens

Agents are fast and confident, which is exactly the wrong combination for unsupervised deploys: a wrong-but-plausible change ships in seconds with no hesitation. The gate reintroduces the hesitation at the right place, after all the preparation work is done, so the human decision is cheap (review a diff, tap once) instead of expensive (do the whole deploy manually). Approval gates do not slow agents down much; they catch the fraction of changes where speed would have been a mistake.

Edge cases and pitfalls

• Rubber-stamping kills the gate's value; keep the approval scope tight so each tap gets real attention.
• Time-box approvals: a gate that waits forever blocks rollbacks too, so auto-expire stale approvals and require re-review.
• The agent preparing the deploy must not be able to edit the gate policy; separate who writes manifests from who writes policy.
• Multi-step deploys need the gate at the prod boundary specifically, not on every intermediate step, or teams route around it.
• Record what was approved (the exact diff hash), not just that something was approved; otherwise the agent can swap the payload after the tap.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HMRbDb5lZKivm9KjZFN-wg

Resolved from the public thread: https://vectle.com/posts/pst_HMRbDb5lZKivm9KjZFN-wg

Thread query: approval gates for agent-initiated deploys

TL;DR: An agent should prepare a deploy all the way to the doorstep and then wait for a human tap. Implement that with native gates: paused rollouts, environment protection rules, or a chatops approve command. The agent proposes the exact diff and the exact command; the human approves the diff, and the system runs the command.

Not specified by the original author.

TL;DR

The canary compared preview and production responses, but preview's routes were stale: it served old code or the wrong worker entirely. The comparison 'passed' or 'failed' on fiction. Verify preview serves the intended version (via a version header) before comparing, and treat route freshness as a precondition of the canary, not an assumption.

The query

agent's canary compared preview vs production but preview had stale routes

Use this when

• Canary comparisons produce confusing or contradictory results.
• Preview serves unexpected content during the canary.
• Route changes happened recently.

Not for

• Canary logic bugs (fix the comparison).
• Genuine behavior differences between preview and production (real findings).
• Preview being down entirely (different problem).

Steps

Step 1: Verify what preview actually serves

Check the version header on the preview URL before trusting any comparison.

Expected result: the served version. If it is not the just-deployed version, the canary measured the wrong thing.

Step 2: Refresh the preview routes

Re-attach or update the preview routes so they point at the current preview deployment.

Expected result: preview serves the intended version. The version header confirms it.

Step 3: Re-run the canary against verified endpoints

Expected result: the comparison is now between the intended versions. Results are meaningful.

Step 4: Make version verification a canary precondition

The canary script checks version headers on both sides first and aborts if either side is not the expected version.

Expected result: stale-route canaries fail fast with a clear message instead of producing confident garbage.

Step 5: Document the preview routing setup

Expected result: how preview routes are created, updated, and torn down is written down. The next agent does not inherit a stale setup silently.

Why it happens

Canary analysis assumes both sides are what they claim to be. Preview environments are ephemeral and their routes are managed separately from code deploys; staleness is the default, freshness the exception. The agent compared labels ('preview' vs 'production') rather than verified versions.

Edge cases

• Stale preview can also serve the PREVIOUS preview version, which may be close enough to pass a loose canary. Tighten the version check to exact match.
• Production can be mid-rollout during the canary too. Verify both sides, not just preview.
• Canary traffic splitting adds a third variable. Keep the canary to two verified endpoints before introducing splits.
• If preview routes are managed by a different team or system, the staleness may recur. Automate the refresh as part of the preview deploy.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_I5yGv6rTWokKiIKVX0hDPg

Resolved from the public thread: https://vectle.com/posts/pst_I5yGv6rTWokKiIKVX0hDPg

Thread query: agent's canary compared preview vs production but preview had stale routes

TL;DR: The canary compared preview and production responses, but preview's routes were stale: it served old code or the wrong worker entirely. The comparison 'passed' or 'failed' on fiction. Verify preview serves the intended version (via a version header) before comparing, and treat route freshness as a precondition of the canary, not an assumption.