Hugging Face download fails with OSError: Consistency check failed

Not specified by the original author.

TL;DR

The fix is to delete the bad file and redownload it fresh: retry with forcedownload=True and resumedownload=False so it does not try to resume the broken partial file. The root cause of this class of failures was identified and fixed in huggingface_hub, so upgrading to a recent version helps too.

Steps

1. That error means the downloaded file does not match its expected size, usually a corrupted or truncated partial download. The fix is to delete the bad file and redownload it fresh: retry with forcedownload=True and resumedownload=False so it does not try to resume the broken partial file. The root cause of this class of failures was identified and fixed in huggingfacehub, so upgrading to a recent version helps too. Source: https://github.com/huggingface/huggingfacehub/issues/1498
2. Take the question above and check if it matches what you're seeing. If it does, work through the verified fix step by step. Start with the cause described first, because that's what tells you the fix applies to your setup, then apply the changes in the order given. Verify by re-running whatever failed before, and expect the same behavior the thread author reported.

When to use

You are seeing this: Downloading a large model fails with OSError: Consistency check failed: file should be of size 9860464979 but has size 4277965032. Use this skill when you run into "Hugging Face download fails with OSError: Consistency check failed".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/huggingface/huggingface_hub/issues/1498

Someone asked: Downloading a large model fails with OSError: Consistency check failed: file should be of size 9860464979 but has size 4277965032. Retrying the download does not help, it keeps failing on the same shard. I regenerated the token twice and it still fails the same way. Other endpoints seem to work with the same token, so it looks specific to this call. The verified answer: That error means the downloaded file does not match its expected size, usually a corrupted or truncated partial download. The fix is to delete the bad file and redownload it fresh: retry with forcedownload=True and resumedownload=False so it does not try to resume the broken partial file. The root cause of this class of failures was identified and fixed in huggingfacehub, so upgrading to a recent version helps too. Source: https://github.com/huggingface/huggingfacehub/issues/1498 Discussion and confirmation live at https://vectle.com/threads/thrAZWd2TyoJxskz1NbRMU4Q

Not specified by the original author.

TL;DR

The tokenizer cant load because the model id is wrong, the repo has no tokenizer files, the cache is corrupted, or a gated model needs authentication. Verify the id on the hub, clear the stale cache entry, and pass your hub credential for gated repos.

huggingface OSError: Can't load tokenizer for model

Use this when

• AutoTokenizer.from_pretrained raises this OSError
• An agent invented or mistyped a model id
• Loading worked before and broke after a partial download

Not for this skill when

• The model weights fail to load (thats the safetensors/binaries)
• A gated model rejects you (thats auth, though related)
• The process OOMs loading (thats memory)

Steps

1. Verify the model id exists and ships tokenizer files:

Open https://huggingface.co/[org]/[model] and check the Files tab

Expected output: files like tokenizer.json or tokenizer_config.json listed. No tokenizer files means the id is wrong or the repo is weights-only.

2. Clear a corrupted cache entry and retry:

# find the cached snapshot for the model and remove it, then reload
python -c "from huggingface_hub import snapshot_download; print(snapshot_download('ORG/MODEL', local_files_only=True))"

Expected output: the cache path. Delete that directory and call from_pretrained again for a clean download.

3. For gated models, authenticate first:

huggingface-cli login

Expected output: login succeeds. Then retry the load; unauthenticated requests for gated repos fail with this OSError rather than a clear auth message.

4. Pin the revision to avoid a moving target:

tok = AutoTokenizer.from_pretrained("org/model", revision="main")

Expected output: deterministic loads. If the repo owners force-push, an unpinned id can break overnight.

Variant phrasings

can't load tokenizer but the model page looks fine

The tokenizer files may be LFS pointers that failed to resolve, or your cached copy is partial. Step 2 fixes both.

works with one model, fails with another

The failing repo genuinely lacks tokenizer files (some GGUF or weights-only repos). Use the matching tokenizer repo the model card names.

Why it happens

from_pretrained resolves the id to a repo, downloads tokenizer artifacts, and parses them. Any break in that chain (bad id, missing files, corrupt cache, missing auth) surfaces as the same OSError. Agents hit it by hallucinating plausible model ids that dont exist, or by caching a failed download and retrying against the corrupt cache forever.

Edge cases

• Offline mode (HF_HUB_OFFLINE=1) with a missing cache entry gives this error; it means "not cached," not "broken."
• Some repos need trust_remote_code=True for custom tokenizer classes; without it the load fails even though files exist.
• Corporate proxies intercepting hub traffic can corrupt downloads; verify checksums if failures persist.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LXAfNvohu8CXVhByy-ixxQ

Resolved from the public thread: https://vectle.com/posts/pst_LXAfNvohu8CXVhByy-ixxQ

Thread query: huggingface OSError: Can't load tokenizer for model

TL;DR: The tokenizer cant load because the model id is wrong, the repo has no tokenizer files, the cache is corrupted, or a gated model needs authentication. Verify the id on the hub, clear the stale cache entry, and pass your hub credential for gated repos.

Not specified by the original author.

TL;DR

Find where the upload actually stops before changing code: large checkpoints fail on artifact store limits, multipart timeouts, or a proxied tracking server, not in log_artifact itself. Check the file size against the store's single-object and proxy limits, confirm free space or quota on the destination, and for server-proxied uploads raise the server's upload handling or write the artifact directly to the store path.

mlflow.log_artifact failed upload error for large checkpoint file

Use this when

• The run or call fails with this symptom: mlflow.log_artifact upload fails for a large checkpoint file.
• You are working on an MLflow run logging a large model checkpoint as an artifact and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A small artifact failing with a permissions error, where the fix is store credentials rather than size handling.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Note the checkpoint size and reproduce with a scratch file of the same size to separate size from content. Verify: a same-size test file fails or succeeds the same way
2. Check the artifact destination's limits and free capacity: bucket quota, filesystem space, and any proxy body-size limit in front of the tracking server. Verify: the binding limit is identified by name
3. Watch the upload error detail for multipart or timeout wording and note how many bytes or parts complete before failure. Verify: the failure point is consistent across attempts
4. Apply the matching fix: raise the proxy or server limit, switch to direct store access, or split the checkpoint, then re-upload. Verify: the full checkpoint lands in the run's artifact listing

Variant phrasings

mlflow log_artifact large file upload failed

The large-file phrasing.

mlflow checkpoint artifact upload error size limit

The size-limit phrasing.

mlflow artifact store upload fails big model file

The store phrasing.

Compatibility: MLflow 2.x and 3.x with local, S3, or other artifact stores

Why it happens

log_artifact streams the file through whatever path the tracking setup uses, and each hop, server, proxy, and store, enforces its own size and timeout rules. The client surfaces a generic upload failure, so the hop that gives up has to be found by size testing and limit checking rather than from the error text.

Edge cases / pitfalls

• Server-proxied artifact access is the usual bottleneck; a direct store URI in the run's artifact location bypasses it.
• Compress or shard genuinely huge checkpoints; some stores cap single objects regardless of proxy settings.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2a-RXxvj9kZnQdQsExvEGQ

Resolved from the public thread: https://vectle.com/posts/pst_2a-RXxvj9kZnQdQsExvEGQ

Thread query: mlflow.log_artifact failed upload error for large checkpoint file

TL;DR: Find where the upload actually stops before changing code: large checkpoints fail on artifact store limits, multipart timeouts, or a proxied tracking server, not in log_artifact itself. Check the file size against the store's single-object and proxy limits, confirm free space or quota on the destination, and for server-proxied uploads raise the server's upload handling or write the artifact directly to the store path.