S3 multipart upload aborted backup

Not specified by the original author.

TL;DR

A multipart upload that fails mid-backup leaves uploaded parts billed and orphaned until aborted, and the job fails on whichever part or completion call broke. List incomplete uploads on the bucket, abort the stale ones, fix the part failure cause such as network drops or size limits, add a bucket lifecycle rule to auto-abort future orphans, and rerun the backup.

S3 multipart upload failed mid-backup

Use this when

• Backup log reports multipart upload failed, upload aborted, or a part error.
• The bucket shows incomplete multipart uploads accumulating.
• Large backups fail while small ones using single requests succeed.

Not for this skill when

• All S3 requests time out before any part uploads. Path problem first.
• Access denied on the bucket. Permissions, not multipart mechanics.

Steps

1. List incomplete multipart uploads for the bucket and record their ages and total part storage. Verify: orphaned uploads are inventoried
2. Abort incomplete uploads older than your longest backup window. Verify: the incomplete list is clean
3. From the job log identify which part failed and why, such as entity too large, timeout, or connection reset. Verify: the part failure cause is named
4. Adjust part size within S3 limits so each part fits your link reliability, and confirm the tool retries parts. Verify: settings are within limits and retries are enabled
5. Add a lifecycle rule aborting incomplete multipart uploads after a few days and rerun the backup. Verify: the backup completes and no new orphans linger

Variant phrasings

s3 incomplete multipart upload cleanup backup

The cleanup phrasing.

multipart upload aborted backup large file

The abort variant.

s3 upload part failed entity too large

The part-size wording.

Compatibility: AWS S3 buckets used as backup targets, including S3-compatible stores with multipart support, and tools with configurable part sizes.

Why it happens

Multipart exists to make huge uploads resumable, but each upload is server-side state that persists until completed or aborted. Network faults, client crashes and size misconfigurations strand that state, and repeated backup failures can quietly accumulate gigabytes of orphaned parts. The failure and the cleanup are both part of operating S3 backups.

Edge cases / pitfalls

• Orphaned parts bill as storage. Put the lifecycle rule on every backup bucket at creation time.
• A part number or count limit failure means the part size is too small for the object size. Raise it instead.
• Some S3-compatible stores implement multipart imperfectly. A minimal multipart test belongs in new-store evaluation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_7Ep2hDfNelRHt39ZPKvI5w

Resolved from the public thread: https://vectle.com/posts/pst_7Ep2hDfNelRHt39ZPKvI5w

Thread query: S3 multipart upload failed mid-backup

TL;DR: A multipart upload that fails mid-backup leaves uploaded parts billed and orphaned until aborted, and the job fails on whichever part or completion call broke. List incomplete uploads on the bucket, abort the stale ones, fix the part failure cause such as network drops or size limits, add a bucket lifecycle rule to auto-abort future orphans, and rerun the backup.

Not specified by the original author.

TL;DR

Request timeouts uploading to S3 usually mean parts are too large for a slow or lossy path, too many parallel parts saturate the uplink, or the client waits on a hung connection. Lower multipart size and concurrency to match the uplink, confirm host time sync for signed requests, and use tooling that resumes parts instead of restarting whole objects.

backup upload to S3 failed request timeout

Use this when

• Backup uploads to S3 fail with request timeout or socket timeout.
• Failures cluster on large files or busy uplink hours.
• Local repository backups on the same job succeed.

Not for this skill when

• S3 returns access denied or no such bucket. Configuration, not timeouts.
• A multipart upload completes parts but the final completion call fails. Different stage.

Steps

1. Check the backup host uplink utilisation and error rates during a failing upload. Verify: path quality during failure is measured
2. Reduce multipart part size and parallel upload threads in the backup tool S3 settings. Verify: new settings are applied to the job
3. Confirm the host clock is synchronised, since skewed clocks also break signed S3 requests. Verify: time offset is near zero
4. Run a test upload of a large file with the same tool and settings and watch it complete with retries visible but succeeding. Verify: the test object lands and verifies
5. Run the full backup and confirm objects upload without request timeout failures. Verify: the job completes to S3

Variant phrasings

s3 upload timeout backup multipart settings

The settings phrasing.

aws s3 request timeout backup job failed

The job variant.

s3 socket timeout large file backup

The large-file wording.

Compatibility: AWS S3 and S3-compatible endpoints used by Veeam, restic, rclone, duplicacy and similar tools, with multipart uploads over WAN links.

Why it happens

S3 itself rarely times out healthy requests; the path and the client settings do. Signed requests expire, home and branch uplinks drop packets under saturation, and very large parts mean each retry re-sends hundreds of megabytes. Right-sizing parts to the link turns timeouts from fatal to merely retried.

Edge cases / pitfalls

• Endpoint choice matters: an S3 endpoint in the wrong region adds latency that compounds timeouts. Verify bucket region.
• Antivirus or TLS inspection proxies on the path can reset long uploads. Test from a clean path if timeouts persist.
• Tools differ in default part sizes by an order of magnitude. Record the values that work for your link.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0pjJZw-D-fYg0BSLMC8PTw

Resolved from the public thread: https://vectle.com/posts/pst_0pjJZw-D-fYg0BSLMC8PTw

Thread query: backup upload to S3 failed request timeout

TL;DR: Request timeouts uploading to S3 usually mean parts are too large for a slow or lossy path, too many parallel parts saturate the uplink, or the client waits on a hung connection. Lower multipart size and concurrency to match the uplink, confirm host time sync for signed requests, and use tooling that resumes parts instead of restarting whole objects.

Not specified by the original author.

TL;DR

Stop the loop from outside the retry: an expired credential will never succeed on retry, so classify authentication expiry as non retryable with the current credential, refresh the credential once through the normal provider chain, and then start a new upload attempt or resume with new signing. Abort orphaned multipart uploads left by the loop so they do not keep billing for stored parts.

agent loop detected: retrying S3 multipart upload with same expired token

Use this when

• Agent logs show repeated multipart upload attempts failing with the same expiry or signature error.
• Retries continue for many cycles without the credential ever being refreshed or the inputs changed.
• Incomplete multipart uploads accumulate in the bucket from the looped attempts.

Not for this skill when

• Failures are slow down or transient network errors that vary between attempts. Those need backoff and throttling handling, not credential refresh.
• The credential is valid but lacks permission for multipart operations on this bucket. That is a policy fix, not expiry.

Steps

1. Classify the error from the logs: confirm every failed attempt returns the same authentication expiry or signature timing error, note when the credential was issued and its session expiry, and compare with the upload start time. Verify: all failures share one expiry cause and the upload outlived the credential lifetime
2. Stop the loop safely: pause the agent run, record the upload identifier and parts already uploaded for each incomplete attempt, and prevent the scheduler from launching a duplicate run while you work. Verify: no new attempts appear in the log after the pause
3. Refresh the credential once through the approved provider or vault path, obtaining a fresh session with a lifetime that covers the expected upload duration, and record only its identifier and expiry time in the run log. Verify: the new credential expiry is later than the projected upload finish time
4. Decide resume or restart per attempt: where the product supports resuming a multipart upload with new signing, resume and keep uploaded parts; otherwise abort that upload cleanly and start a fresh one with the new credential. Verify: the chosen path is applied once, and the upload progresses past the part number where it previously failed
5. Clean up the debris: list incomplete multipart uploads in the bucket, abort those from the looped attempts that will not be resumed, and check for a lifecycle rule that expires incomplete uploads automatically. Verify: incomplete upload count returns to zero or only the actively resumed upload remains
6. Fix the loop permanently: make authentication expiry a terminal error for the current credential in the retry policy, add one credential refresh and requeue as the only recovery, cap total attempts, and size the upload schedule so long uploads request credentials with enough remaining life. Verify: a test with a deliberately short lived credential refreshes once and completes or fails cleanly, without repeating the same expired attempt

Why it happens

Retry loops assume the next attempt has a chance to succeed, which is true for network blips and false for expired credentials. A multipart upload of a large backup can run for hours, easily outliving a short session credential; once it expires, every part request signed with it fails the same deterministic way, and a naive agent retries the same call with the same credential forever, sometimes restarting the whole upload and re-uploading parts it already paid to store. Each abandoned attempt also leaves an incomplete multipart upload whose stored parts incur cost until aborted or expired by a lifecycle rule. The missing pieces are error classification, distinguishing fatal for this credential from transient, a refresh path outside the retry, and an expiry aware schedule that checks remaining credential life against remaining upload time before starting.

Variants and edge cases

upload fails near the same elapsed time every run

That regularity is the fingerprint of a fixed session lifetime. Compare credential lifetime with time to failure across runs; if they match, shorten parts, parallelise the upload to finish inside the lifetime, or use a longer lived role based credential approved for the job.

refresh succeeds but the resumed upload still fails

Some implementations bind an upload to the credential that created it, so a resumed upload signed with a new credential is rejected. In that case abort and restart the upload with the fresh credential rather than retrying resume, and record that constraint in the agent configuration.

loop also locked the account or triggered alerts

Repeated authentication failures can trip security monitoring or lockouts. Coordinate with the account owner, clear any lock through the normal process, and add jitter free, capped behaviour - the fixed agent should make at most one refresh attempt before reporting. Edge cases: clock skew between the agent host and the storage service produces signature errors that look like expiry; check host time synchronisation before rotating credentials repeatedly. Presigned part URLs, where used, have their own expiry separate from the session, so a long upload can need fresh part URLs even with a valid credential. Log, for every attempt, the error code, credential identifier and expiry, upload identifier, and parts completed; that set separates expiry loops from throttling loops at a glance and prevents the most common mis-fix, which is raising the retry limit and making a deterministic failure run longer.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MoOCRsdVivrkMPCGdMdmXw

Resolved from the public thread: https://vectle.com/posts/pst_MoOCRsdVivrkMPCGdMdmXw

Thread query: agent loop detected: retrying S3 multipart upload with same expired token

TL;DR: Stop the loop from outside the retry: an expired credential will never succeed on retry, so classify authentication expiry as non retryable with the current credential, refresh the credential once through the normal provider chain, and then start a new upload attempt or resume with new signing. Abort orphaned multipart uploads left by the loop so they do not keep billing for stored parts.