cloud backup throttled uploads never finish before window ends

TL;DR

Do the window arithmetic: data to upload divided by the throttled effective rate tells you whether the job can ever finish in the window. Raise effective throughput by fixing the throttling source, increasing parallelism and part size, and seeding or splitting the job, instead of simply extending the window every week. Measure the rate the provider actually allows, not the line rate you pay for.

cloud backup throttled uploads never finish before window ends

Use this when

• Cloud backup jobs are still uploading when the window closes, repeatedly.
• Provider logs or responses show slow down, rate limit, or request throttling messages.
• Local backup completes but the cloud copy of the same job never catches up.

Not for this skill when

• Uploads fail immediately with access denied, signature, or endpoint errors. That is configuration, not throttling.
• The job fits easily in the window but hangs on one file. That is a stall, not a rate problem.

Steps

1. Quantify the gap: bytes the job must upload, window length, and the average and peak throughput actually achieved in the logs for the last five runs. Verify: required rate and achieved rate are both written down and the shortfall is explicit
2. Identify the throttling source from provider responses and account limits: request rate per prefix or bucket, account level bandwidth caps, backup product concurrency limits, and any provider fair use policy. Note retry and backoff entries in the job log. Verify: throttling evidence, such as slow down responses or capped request counts, is tied to a specific limit
3. Raise parallelism correctly: increase concurrent parts and transfers for large objects, set a multipart part size that keeps enough parts in flight without creating millions of small requests, and confirm the host has the CPU and memory to sustain it. Verify: achieved throughput rises in a test upload of a representative large file outside the production window
4. Reduce what must move in the window: enable changed block or incremental forever transfer, compress before upload, exclude caches and temporary data, and seed the initial full to the cloud instead of uploading it through the daily window. Verify: daily upload bytes fall to the true changed data volume
5. Split and stagger the fleet: give critical systems the early window, move bulk systems to longer weekend windows, and stop all jobs starting at once and throttling each other against the same account limit. Verify: per job achieved rate rises when jobs no longer overlap fully
6. Negotiate or configure the limit itself where possible: request a provider limit increase with measured request rates, remove stale product side caps, and alert when a job will miss its window based on early progress. Verify: an in progress job projects its finish time and alerts before the window closes

Why it happens

Cloud storage throttles in two ways that both look like a slow network: request rate limits, which punish many small objects and parts, and throughput or account caps, which punish bulk transfer. A backup job tuned for local disk often uses small blocks, low concurrency, and aggressive retries; against an object store that means thousands of requests per minute against one prefix, slow down responses, exponential backoff, and an effective rate a fraction of the line speed. Meanwhile the window was sized when data was smaller, so even an unthrottled job may no longer fit. Retries make it self defeating - as the job falls behind, retry traffic grows and consumes more of the allowed request rate. That is why measuring achieved rate, request rate, and required rate separately matters: each points to a different fix, and only one of them is buying more bandwidth.

Variants and edge cases

throttling starts at the same time every night

All jobs start together and jointly hit an account or prefix limit, or a provider or ISP peak policy applies. Stagger starts, spread objects across prefixes where the provider recommends it, and shift bulk jobs out of the peak hour.

small file workloads throttle, large files do not

Request rate, not bytes, is the limit. Archive small files into larger containers before upload, increase part and block sizes, and reduce per file verification calls during the window, moving verification to a later pass.

uploads slow after hours of good progress

Look for thermal, CPU, or provider burst credit exhaustion, token bucket style limits that allow a fast start then clamp down, and local staging disks that fill and force direct slow paths. Sustained rate, not starting rate, is the number to design around. Edge cases: egress costs and minimum storage durations can make aggressive re-uploads and short lived seeded copies expensive, so model cost alongside speed. Also avoid fixing throttling by disabling integrity checks or verification; a fast upload that cannot be proven restorable trades a window problem for a recovery problem. Keep a per job window budget in the runbook - bytes, allowed request rate, achieved sustained rate, and projected finish - reviewed monthly, because data growth silently converts a comfortable job into a throttled one without any configuration change at all.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_u9sbXM4feVGQqe1qSXAKrw