agent retried restore so many times it locked the backup account

Not specified by the original author.

TL;DR

Account lockout from restore retries means the retry loop treats authentication failures like transient errors and keeps presenting bad or expired credentials until the identity system defends itself. Make authentication failures a hard stop with zero automatic retries, fix the credential, unlock the account through the proper process, and add a run-level auth health check before any restore work begins.

agent retried the restore until it locked the account and the run failed

Use this when

• The backup or cloud account locks during agent runs.
• Auth failures repeat rapidly in logs just before the lockout.
• Other users of the shared account are locked out too.

Not for this skill when

• The account was locked by a human password mistakes. Different actor.
• Restore data errors trigger retries that never touch authentication. Retry policy, separate scope.

Steps

1. Confirm from identity logs that the agent attempts caused the lockout threshold to trip. Verify: attempt count and timing are evidenced
2. Change the agent so any authentication failure stops the run immediately and alerts with the credential identity used. Verify: a bad-credential test makes exactly one attempt
3. Repair the underlying credential: refresh, reissue or fix its storage in the vault, and verify it manually once. Verify: a single manual authentication succeeds
4. Unlock the account through the identity owner process and confirm no other automation still hammers it. Verify: the account stays unlocked under observation
5. Add a pre-flight auth check at run start that fails the run cleanly before restore attempts. Verify: the full restore run now passes with healthy auth

Variant phrasings

account locked agent retry authentication storm

The storm phrasing.

agent hammering credentials lockout stop retry

The behaviour variant.

auth failure hard stop no retry automation

The policy wording.

Compatibility: Agents authenticating to backup products, cloud APIs and SSH hosts with lockout-enforcing identity providers and directory services.

Why it happens

Lockout policies assume a human who will stop after a few failures and call for help. An agent loop has no embarrassment and infinite patience, so it converts a stale credential into a denial of service against its own account, and any humans sharing it. Hard-stop semantics for auth errors are a basic safety property of automation, not an optimisation.

Edge cases / pitfalls

• Shared accounts multiply the blast radius. Agents should hold their own identities precisely so lockouts stay contained.
• Some systems lock on attempts across all sources. Coordinate with other automations before blaming one agent.
• Alert on the first auth failure, not the lockout. The lockout means the alerting already failed once.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2cKd2FdvwDgL8Q-gAE-lrg

Resolved from the public thread: https://vectle.com/posts/pst_2cKd2FdvwDgL8Q-gAE-lrg

Thread query: agent retried the restore until it locked the account and the run failed

TL;DR: Account lockout from restore retries means the retry loop treats authentication failures like transient errors and keeps presenting bad or expired credentials until the identity system defends itself. Make authentication failures a hard stop with zero automatic retries, fix the credential, unlock the account through the proper process, and add a run-level auth health check before any restore work begins.

Not specified by the original author.

TL;DR

A held lock is information, not a reason to spin. On the first lock error, read the lock details to identify the owning host, process, and lock age, decide whether it is live or stale, and if it is live, back off on a capped schedule tied to the remaining backup window and stop early enough to alert and reschedule. Only clear a lock through the repository unlock path after proving the owner is gone, and never let retry behaviour consume the window that a rescheduled run could still use.

agent retried a locked restic repository until the backup window closed

Use this when

• Backup logs show repeated repository locked messages for hours, ending with the window closing and the backup skipped.
• Another backup, check, or prune job on the same repository overlaps the scheduled run.
• The agent reports the backup as failed for a lock, but no alert was raised until the window had already ended.

Not for this skill when

• The repository check reports damaged packs or an index problem.
• No lock exists and the failure is a network or credential error reaching the repository.

Steps

1. Read the lock before retrying anything. On a lock error, list the repository locks and record the owning host, user, process id if shown, creation time, and whether the lock is exclusive. This single reading determines the whole strategy, because a live exclusive lock from a running prune is a scheduling conflict while an hours old lock from a dead host is cleanup. Verify: the run log names the lock owner and age on the first occurrence, not after retries
2. Decide live versus stale with evidence. Check whether the owning host and process are still active where that is possible, compare lock age against the longest legitimate operation for this repository, and treat a lock as stale only when the owner is confirmed gone or unreachable past that bound. If liveness cannot be determined, treat the lock as live. Verify: the run records a live or stale verdict with the evidence used
3. Clear only proven stale locks, through the proper path. Use the repository unlock command for the stale lock, then run a quick repository metadata read to confirm access, and record the removed lock id and the evidence. Never delete lock files directly from the storage backend, which can race a live owner and corrupt an index update. Verify: after unlocking, a read-only repository command succeeds and the action is in the run log
4. Back off against the window, not forever. If the lock is live, wait with exponential backoff starting at a few minutes, cap each wait, and set a hard stop at the point where the remaining window can no longer contain the backup itself plus a margin. Poll the lock state at each wait rather than blindly invoking the backup. Verify: the log shows waits, lock rechecks, and a stop decision made while usable window still remained
5. Exit early and reschedule with context. When the stop point is reached, fail fast with a lock conflict verdict naming the owner and its operation, raise the alert immediately, and schedule a follow-up run after the conflicting operation normally ends. Include the lock details so an operator can act without re-diagnosing. Verify: the alert arrives before the window closes and a follow-up run is on the schedule
6. Remove the overlap at the source. Review the schedules of backups, checks, and prunes sharing the repository, add a harness level single-flight guard per repository so two of this agent jobs never contend, and stagger third party jobs where possible. Record the guard in the job configuration. Verify: over the following week no run reports a lock held by another of the agent own jobs

Why it happens

Restic locks exist to stop two writers corrupting shared repository state, so a lock usually means a legitimate operation such as a long prune is running, or a crashed job left one behind. A naive retry loop treats it as transient noise, retries on a short interval, and quietly eats the entire window, converting a delay into a missed backup. Reading the lock and reasoning about its owner turns the response into either a safe unlock or an early, visible reschedule.

Variants and edge cases

Edge cases: prune and forget operations can hold exclusive locks for a very long time on large repositories, so the legitimate duration bound must come from observed history, not a guess. A lock from the same host with a reused process id can look alive when it is not; combine process, host, and age evidence. Network storage can leave a lock looking stale during a partition while the owner is still writing, which is another reason direct backend deletion of lock files is off limits.

restic backup skipped, repository locked all night

Use when lock messages fill the log until the window ends. Identify the owner on first occurrence next time and set a window aware stop.

agent kept retrying a locked repository

Use when retry count is high and waits are short. Switch to lock state polling with capped backoff.

stale restic lock after a crashed backup

Use when the owner host or process is confirmed gone. Clear it through the unlock command and verify with a read.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Tkfd1JhkuYtRFhuecj4LjA

Resolved from the public thread: https://vectle.com/posts/pst_Tkfd1JhkuYtRFhuecj4LjA

Thread query: agent retried a locked restic repository until the backup window closed

TL;DR: A held lock is information, not a reason to spin. On the first lock error, read the lock details to identify the owning host, process, and lock age, decide whether it is live or stale, and if it is live, back off on a capped schedule tied to the remaining backup window and stop early enough to alert and reschedule. Only clear a lock through the repository unlock path after proving the owner is gone, and never let retry behaviour consume the window that a rescheduled run could still use.

Not specified by the original author.

TL;DR

Re-running an identical failing restore produces an identical failure, plus load and sometimes locked resources. Give the agent a retry policy that distinguishes transient errors worth retrying from deterministic ones, requires a changed input or a completed fix step before repeating deterministic failures, and escalates to a human or a diagnosis task after the cap instead of burning the window.

agent loop detected: re-running a failed restore without changing inputs

Use this when

• Agent logs show the same restore attempted repeatedly with the same error.
• Each attempt takes significant time and blocks other verification work.
• No diagnostic step happens between attempts.

Not for this skill when

• Retries with backoff succeed on transient network errors. That is healthy retry behaviour.
• A restore is repeated deliberately after an operator fixes storage. Changed input, intended.

Steps

1. Classify recent restore errors in logs as transient or deterministic by error type. Verify: each repeated failure is classified
2. Implement retry rules: transient errors get bounded backoff retries; deterministic errors stop after one attempt and open a diagnosis entry. Verify: a simulated deterministic failure stops after one run
3. Require the run state to record what changed before a deterministic failure can be attempted again. Verify: an unchanged retry request is refused by the agent
4. Route capped failures to the alerting path with full context. Verify: an escalation is created in a test
5. Run the verification suite with a seeded bad snapshot. Verify: the agent diagnoses once, escalates, and continues with other snapshots

Variant phrasings

agent retry deterministic failure no change loop

The policy phrasing.

failed restore repeated identical attempts stop

The symptom variant.

retry policy transient vs deterministic errors agent

The classification wording.

Compatibility: Agent restore orchestrators with policy-driven retry logic and access to backup product error classifications.

Why it happens

Naive retry logic treats every failure as weather: wait and try again. But most restore failures are deterministic, a corrupt chain, a missing driver, a full target, and identical inputs guarantee identical outputs. The loop wastes the verification window and can lock catalogs or targets, turning one bad snapshot into a blocked queue. Classification and change-gating are the fix.

Edge cases / pitfalls

• Some errors are transient only sometimes, such as timeouts under load. Cap their retries too and record load context.
• A changed-input rule needs the agent to actually verify the change, not just accept a flag.
• Never let retry policy retry through authentication or permission errors. Those need humans.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_s5GZFrUxNerjDVRbRjMSKA

Resolved from the public thread: https://vectle.com/posts/pst_s5GZFrUxNerjDVRbRjMSKA

Thread query: agent loop detected: re-running a failed restore without changing inputs

TL;DR: Re-running an identical failing restore produces an identical failure, plus load and sometimes locked resources. Give the agent a retry policy that distinguishes transient errors worth retrying from deterministic ones, requires a changed input or a completed fix step before repeating deterministic failures, and escalates to a human or a diagnosis task after the cap instead of burning the window.