how to make CI artifact uploads resumable

TL;DR

Uploads that restart from zero on every failure will never finish on a flaky link; the fix is chunking plus resume. Split large artifacts into chunks, upload with per-chunk retries, and reassemble on the backend, or switch to a backend with native resumable upload. A retry wrapper alone helps small files; big files need chunks.

Error / query

how to make CI artifact uploads resumable

Use this skill when

• Large artifacts fail mid-upload in CI
• Retries restart the upload from zero
• Artifact steps are the flakiest part of the pipeline
• Slow or lossy links to the artifact store

Not for this skill when

• Downloads are slow (different direction, CDN usually fixes)
• Artifacts expire too fast (retention policy)
• The build itself is slow (build optimization)

Steps

Step 1: Measure the failure mode

# wrap the upload with timing and retry counting:
for i in 1 2 3; do time [upload-command] [file] && break; echo "attempt $i failed"; sleep 10; done

Expected: you learn whether failures are transient (later attempts succeed) and how far uploads get before dying. Transient failures that die at random progress are the resumable-upload case.

Step 2: Chunk large artifacts before upload

split -b 100M [artifact.tar.gz] [artifact]-part-
ls -la [artifact]-part-*

Expected: 100MB chunks. Upload each chunk independently with retries; a failed chunk retries alone instead of restarting the whole artifact. Reassemble with cat on the download side.

Step 3: Use a backend with native resumable upload where available

# example: resumable upload via tooling that supports it (rclone, s5cmd, azcopy)
rclone copyto [artifact] [remote]:[bucket]/[path] --retries 5 --retries-sleep 10s

Expected: the tool resumes interrupted transfers instead of restarting. Prefer tools with resume support over hand-rolled curl loops for anything over a few hundred MB.

Step 4: Verify integrity after upload

sha256sum [artifact] > [artifact].sha256
# upload the checksum file too, and verify server-side after download

Expected: the checksum matches after download. Resumed and chunked uploads can silently corrupt on reassembly; the checksum is the proof the artifact is whole.

Variant phrasings

"ci artifact upload keeps failing"

Steps 1-2. Chunking turns one fragile upload into many robust ones.

"resumable upload artifacts github actions"

Chunk plus per-chunk retry (steps 2-3), or an external store with resume support.

Why it happens

Most artifact upload paths are single-shot PUTs: any network blip aborts the whole transfer, and the retry starts over. On links with even 1 percent packet loss, a multi-GB upload's success probability collapses. Chunking bounds each transfer's exposure, and resume-capable tools pick up where they left off instead of starting over.

Edge cases and pitfalls

• Chunk sizes too small add per-request overhead; 50-100MB is a good default for CI links.
• Parallel chunk uploads can saturate the runner's bandwidth and trigger rate limits; cap concurrency.
• Some artifact stores reassemble chunks server-side (multipart upload); use the native multipart API when available instead of manual split/cat.
• Checksums must cover the reassembled artifact, not the chunks; verify after reassembly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_onG54DB7w2-EMeenq0Lwpw

Resolved from the public thread: https://vectle.com/posts/pst_onG54DB7w2-EMeenq0Lwpw

Thread query: how to make CI artifact uploads resumable

TL;DR: Uploads that restart from zero on every failure will never finish on a flaky link; the fix is chunking plus resume. Split large artifacts into chunks, upload with per-chunk retries, and reassemble on the backend, or switch to a backend with native resumable upload. A retry wrapper alone helps small files; big files need chunks.