agent kept retrying a failed deploy: how to add circuit breakers to agent loops

TL;DR

An agent that retries a failed deploy forever turns a bad rollout into an outage amplifier: each retry re-applies, restarts pods, and pages everyone again. Put a circuit breaker on the agent's retry loop: count consecutive failures, stop after a small threshold, and require a changed input before the next attempt. Retries should converge, not spin.

attempt 1 failed, retrying...
attempt 2 failed, retrying...
attempt 47 failed, retrying...

Steps

1. Give the retry loop a hard stop: max attempts and a consecutive-failure counter.

MAX_ATTEMPTS=3
ATTEMPT=1
while [ $ATTEMPT -le $MAX_ATTEMPTS ]; do
  deploy_step || ATTEMPT=$((ATTEMPT+1))
done

Expected: the loop exits after 3 failures instead of running forever. The exact number matters less than the fact that a limit exists and is enforced.

2. Add exponential backoff with jitter between attempts so retries do not hammer the API server.

Expected: waits of roughly 5s, 15s, 45s between attempts. The jitter keeps multiple agents from retrying in lockstep and thundering the same endpoint.

3. Require a changed input before any retry past the first: a new image tag, a config fix, or a human-confirmed state change.

Expected: the agent cannot retry the identical failing command twice. It must either fix something or escalate, which is the behavior that actually resolves incidents.

4. On breaker trip, the agent stops, summarizes what failed with the evidence, and pages a human instead of looping.

Expected: one clear escalation message with the last error, the attempts made, and what changed between them, instead of 47 identical log lines. The incident gets a human while the blast radius is still small.

When to use

• An agent retries deploys, restarts, or remediations in a loop without making progress.
• You are writing the control loop for any agent that takes mutating actions.
• A past incident got worse because automation kept re-applying a broken change.

When not to use

• The operation is idempotent and safe to repeat, like a read-only health check (those can poll; this is about mutating retries).
• A human is in the loop approving each attempt (the human is the breaker).
• The retry is against a transient network blip with a known-good success rate (short bounded retries are fine there).

Why it happens

Agent loops are usually written as "try, and if it fails, try again" with no memory of past attempts, because each iteration feels like a fresh decision to the model. But the environment is not fresh: the same broken image, the same bad config, the same failing probe. Without a failure counter and a demand for changed inputs, the loop cannot distinguish "transient, try again" from "structurally broken, stop", so it picks "try again" forever.

Edge cases

• Backoff alone is not a breaker; sleeping 60 seconds between infinite retries just makes the loop slower, not safer.
• Distinguish retryable from fatal errors: ImagePullBackOff on a bad tag never self-heals, so fail fast instead of burning all attempts.
• Multiple agents watching the same deploy each need the breaker, or one agent's stop just hands the loop to another; coordinate through a shared attempt counter or a lock.
• Log every attempt with its input hash so the postmortem can see whether the agent actually changed anything between retries.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0tH5F1kIiT5jeZa3AjXvnQ