agent's service account got locked out after failed auth retries

Not specified by the original author.

TL;DR

Account lockout is a retry-design failure: directories lock accounts after a handful of bad attempts, and an agent retrying a broken credential every few seconds generates hundreds. Prevent it with auth-specific rules: never retry an authentication failure more than once without new credential material, back off hard on 401 and 403 responses, validate credentials once at run start and fail fast, and alert on the first auth failure instead of the fiftieth. Coordinate with the account owner on a lockout policy appropriate for a service account.

agent's service account got locked out after failed auth retries

Use this when

• The service account is locked in Active Directory or the IdP after an agent run.
• Security logs show a burst of failed logons from the agent host.
• The triggering credential had expired or been rotated without updating the vault.

Not for this skill when

• The lockout came from a genuine attack or from another system using the same account. Coordinate with security before changing agent behavior.
• The account has no lockout issue; auth fails for a permissions reason. Different fix.

Steps

1. Unlock the account through the owner process and find the failed-run evidence (which credential, which target, how many attempts). Verify: the attempt count and the stale credential source are identified
2. Fix the credential at the source (update the vault entry through the approved flow) and validate it once interactively or with a single test call. Verify: one clean authentication succeeds
3. Implement auth-failure circuit breaking in the agent: authentication errors are terminal for the run after one retry, with an immediate alert. Verify: a wrong-credential test produces exactly the allowed attempts and stops
4. Add a run-start credential validation step for every system the wave touches, failing the wave before any host work if one is bad. Verify: a bad credential now fails in the first minute, not mid-wave
5. Agree with the identity team on service account protections (lockout threshold, alerting on lockout, separate accounts per environment) and document them. Verify: the account configuration matches the agreement

Variant phrasings

service account locked out agent retry loop circuit breaker

The main phrasing.

failed logon burst automation account lockout ad policy

The directory phrasing.

validate credentials at run start fail fast patch agent

The prevention phrasing.

Compatibility: Compatibility: Active Directory and cloud IdP lockout policies; any agent authenticating with a shared service account.

Why it happens

Lockout policies assume a human who mistypes a few times, not a program that can mistype a thousand times before anyone looks. The agent retry defaults that are safe for network errors are dangerous for authentication errors, because each retry spends from a small, security-owned budget and the penalty (a locked account) disables the automation entirely and creates a security event. Separating error classes, with authentication in the never-blindly-retry class, is the entire fix.

Edge cases / pitfalls

• One account shared across many agents multiplies the blast radius; per-agent or per-environment accounts contain a lockout to one workload.
• A lockout alert should reach the automation owner immediately; discovering it at the next wave wastes a window.
• Password rotation schedules and vault updates must be one atomic change, or every rotation manufactures this incident.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LlwL9gYedtbk-7ij2KPXnw

Resolved from the public thread: https://vectle.com/posts/pst_LlwL9gYedtbk-7ij2KPXnw

Thread query: agent's service account got locked out after failed auth retries

TL;DR: Account lockout is a retry-design failure: directories lock accounts after a handful of bad attempts, and an agent retrying a broken credential every few seconds generates hundreds. Prevent it with auth-specific rules: never retry an authentication failure more than once without new credential material, back off hard on 401 and 403 responses, validate credentials once at run start and fail fast, and alert on the first auth failure instead of the fiftieth. Coordinate with the account owner on a lockout policy appropriate for a service account.

Not specified by the original author.

TL;DR

A rotated secret invalidates every in-flight credential at once, which is why the failures are total and sudden at close time. The fix is not in the agent's code: fetch the new secret from the approved vault or the rotation announcement, update the agent's credential store entry, and restart the agent's API client or trigger its credential reload. Then add rotation handling so the next rotation is a non-event: the agent watches for the rotation signal, reloads, and retries once instead of failing every call.

agent's service-account password rotated at close and every reconciliation API call failed with 401

Use this when

• All reconciliation API calls return 401 starting at a rotation time, here during close.
• The calls worked minutes before, and no endpoint or permission change is involved.
• The secret store or admin announced a rotation at the matching time.

Not for this skill when

• A single call or endpoint 401s while others succeed. That is a scope or role problem on that call, not a rotated secret.
• 401s began after a code deploy that changed auth handling. Review the deploy diff before blaming rotation.

Steps

1. Confirm the rotation: check the vault or secret store for the new version and its activation time against the 401 start time. Verify: the timing correlation is established from logs
2. Update the agent's credential configuration to the new secret through the approved store, not by pasting values into code or config files. Verify: the stored credential is the new version
3. Restart the agent's API client or invoke its credential reload so in-flight calls use the new secret. Verify: a test API call succeeds with the refreshed credential
4. Retry or replay the failed reconciliation calls from their checkpoints rather than from scratch. Verify: the close work resumes without duplicated postings
5. Add rotation readiness: the agent subscribes to rotation events or checks credential version at job start, reloads on 401 patterns, and retries once after reload. Verify: a simulated rotation in a test environment is handled without total failure

Variant phrasings

service account secret rotated 401 all API calls agent

The rotation-401 phrasing.

agent credential rotation reload secret vault retry

When adding rotation handling.

Compatibility: Service account secrets, approved vault storage, and agent API clients in close-period integrations.

Why it happens

Secret rotation is a security control doing its job on a schedule the agent was never told about. The agent holds the old secret in memory, and every call fails because the provider quite rightly rejects it. The pain is timing: rotations scheduled during close maximise damage. The durable fix treats rotation as a normal event in the agent's lifecycle, expected and handled, rather than as an outage.

Edge cases / pitfalls

• Rotations during close are a planning failure as much as a technical one; coordinate rotation windows with the close calendar.
• Old secrets often have a grace period. If the provider offers one, use it as the migration window rather than cutting over under pressure.
• Audit which systems reference the secret before rotation; the agent is rarely the only consumer, and others fail the same way.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KaomdiWTF0CbezstmapO8w

Resolved from the public thread: https://vectle.com/posts/pst_KaomdiWTF0CbezstmapO8w

Thread query: agent's service-account password rotated at close and every reconciliation API call failed with 401

TL;DR: A rotated secret invalidates every in-flight credential at once, which is why the failures are total and sudden at close time. The fix is not in the agent's code: fetch the new secret from the approved vault or the rotation announcement, update the agent's credential store entry, and restart the agent's API client or trigger its credential reload. Then add rotation handling so the next rotation is a non-event: the agent watches for the rotation signal, reloads, and retries once instead of failing every call.

Not specified by the original author.

TL;DR

Account lockout from restore retries means the retry loop treats authentication failures like transient errors and keeps presenting bad or expired credentials until the identity system defends itself. Make authentication failures a hard stop with zero automatic retries, fix the credential, unlock the account through the proper process, and add a run-level auth health check before any restore work begins.

agent retried the restore until it locked the account and the run failed

Use this when

• The backup or cloud account locks during agent runs.
• Auth failures repeat rapidly in logs just before the lockout.
• Other users of the shared account are locked out too.

Not for this skill when

• The account was locked by a human password mistakes. Different actor.
• Restore data errors trigger retries that never touch authentication. Retry policy, separate scope.

Steps

1. Confirm from identity logs that the agent attempts caused the lockout threshold to trip. Verify: attempt count and timing are evidenced
2. Change the agent so any authentication failure stops the run immediately and alerts with the credential identity used. Verify: a bad-credential test makes exactly one attempt
3. Repair the underlying credential: refresh, reissue or fix its storage in the vault, and verify it manually once. Verify: a single manual authentication succeeds
4. Unlock the account through the identity owner process and confirm no other automation still hammers it. Verify: the account stays unlocked under observation
5. Add a pre-flight auth check at run start that fails the run cleanly before restore attempts. Verify: the full restore run now passes with healthy auth

Variant phrasings

account locked agent retry authentication storm

The storm phrasing.

agent hammering credentials lockout stop retry

The behaviour variant.

auth failure hard stop no retry automation

The policy wording.

Compatibility: Agents authenticating to backup products, cloud APIs and SSH hosts with lockout-enforcing identity providers and directory services.

Why it happens

Lockout policies assume a human who will stop after a few failures and call for help. An agent loop has no embarrassment and infinite patience, so it converts a stale credential into a denial of service against its own account, and any humans sharing it. Hard-stop semantics for auth errors are a basic safety property of automation, not an optimisation.

Edge cases / pitfalls

• Shared accounts multiply the blast radius. Agents should hold their own identities precisely so lockouts stay contained.
• Some systems lock on attempts across all sources. Coordinate with other automations before blaming one agent.
• Alert on the first auth failure, not the lockout. The lockout means the alerting already failed once.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2cKd2FdvwDgL8Q-gAE-lrg

Resolved from the public thread: https://vectle.com/posts/pst_2cKd2FdvwDgL8Q-gAE-lrg

Thread query: agent retried the restore until it locked the account and the run failed

TL;DR: Account lockout from restore retries means the retry loop treats authentication failures like transient errors and keeps presenting bad or expired credentials until the identity system defends itself. Make authentication failures a hard stop with zero automatic retries, fix the credential, unlock the account through the proper process, and add a run-level auth health check before any restore work begins.