agent loop detected: retrying S3 multipart upload with same expired token

TL;DR

Stop the loop from outside the retry: an expired credential will never succeed on retry, so classify authentication expiry as non retryable with the current credential, refresh the credential once through the normal provider chain, and then start a new upload attempt or resume with new signing. Abort orphaned multipart uploads left by the loop so they do not keep billing for stored parts.

agent loop detected: retrying S3 multipart upload with same expired token

Use this when

• Agent logs show repeated multipart upload attempts failing with the same expiry or signature error.
• Retries continue for many cycles without the credential ever being refreshed or the inputs changed.
• Incomplete multipart uploads accumulate in the bucket from the looped attempts.

Not for this skill when

• Failures are slow down or transient network errors that vary between attempts. Those need backoff and throttling handling, not credential refresh.
• The credential is valid but lacks permission for multipart operations on this bucket. That is a policy fix, not expiry.

Steps

1. Classify the error from the logs: confirm every failed attempt returns the same authentication expiry or signature timing error, note when the credential was issued and its session expiry, and compare with the upload start time. Verify: all failures share one expiry cause and the upload outlived the credential lifetime
2. Stop the loop safely: pause the agent run, record the upload identifier and parts already uploaded for each incomplete attempt, and prevent the scheduler from launching a duplicate run while you work. Verify: no new attempts appear in the log after the pause
3. Refresh the credential once through the approved provider or vault path, obtaining a fresh session with a lifetime that covers the expected upload duration, and record only its identifier and expiry time in the run log. Verify: the new credential expiry is later than the projected upload finish time
4. Decide resume or restart per attempt: where the product supports resuming a multipart upload with new signing, resume and keep uploaded parts; otherwise abort that upload cleanly and start a fresh one with the new credential. Verify: the chosen path is applied once, and the upload progresses past the part number where it previously failed
5. Clean up the debris: list incomplete multipart uploads in the bucket, abort those from the looped attempts that will not be resumed, and check for a lifecycle rule that expires incomplete uploads automatically. Verify: incomplete upload count returns to zero or only the actively resumed upload remains
6. Fix the loop permanently: make authentication expiry a terminal error for the current credential in the retry policy, add one credential refresh and requeue as the only recovery, cap total attempts, and size the upload schedule so long uploads request credentials with enough remaining life. Verify: a test with a deliberately short lived credential refreshes once and completes or fails cleanly, without repeating the same expired attempt

Why it happens

Retry loops assume the next attempt has a chance to succeed, which is true for network blips and false for expired credentials. A multipart upload of a large backup can run for hours, easily outliving a short session credential; once it expires, every part request signed with it fails the same deterministic way, and a naive agent retries the same call with the same credential forever, sometimes restarting the whole upload and re-uploading parts it already paid to store. Each abandoned attempt also leaves an incomplete multipart upload whose stored parts incur cost until aborted or expired by a lifecycle rule. The missing pieces are error classification, distinguishing fatal for this credential from transient, a refresh path outside the retry, and an expiry aware schedule that checks remaining credential life against remaining upload time before starting.

Variants and edge cases

upload fails near the same elapsed time every run

That regularity is the fingerprint of a fixed session lifetime. Compare credential lifetime with time to failure across runs; if they match, shorten parts, parallelise the upload to finish inside the lifetime, or use a longer lived role based credential approved for the job.

refresh succeeds but the resumed upload still fails

Some implementations bind an upload to the credential that created it, so a resumed upload signed with a new credential is rejected. In that case abort and restart the upload with the fresh credential rather than retrying resume, and record that constraint in the agent configuration.

loop also locked the account or triggered alerts

Repeated authentication failures can trip security monitoring or lockouts. Coordinate with the account owner, clear any lock through the normal process, and add jitter free, capped behaviour - the fixed agent should make at most one refresh attempt before reporting. Edge cases: clock skew between the agent host and the storage service produces signature errors that look like expiry; check host time synchronisation before rotating credentials repeatedly. Presigned part URLs, where used, have their own expiry separate from the session, so a long upload can need fresh part URLs even with a valid credential. Log, for every attempt, the error code, credential identifier and expiry, upload identifier, and parts completed; that set separates expiry loops from throttling loops at a glance and prevents the most common mis-fix, which is raising the retry limit and making a deterministic failure run longer.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MoOCRsdVivrkMPCGdMdmXw