hf download gets stuck on large downloads

TL;DR

The Xet team identified it from logs and shipped a fix in hf-xet 1.5.2, so run pip install hf-xet==1.5.2 or newer and the stuck downloads should go through. hf download on a large dataset always gets stuck after a while.

Steps

1. hf download on a large dataset always gets stuck after a while. The progress bars and the XET log just stop updating. Restarting lets it download a bit more, then it hangs again. Disabling XET did not change anything.
2. This is a known bug in the XET download path that causes large downloads to hang. The Xet team identified it from logs and shipped a fix in hf-xet 1.5.2, so run pip install hf-xet==1.5.2 or newer and the stuck downloads should go through. A few people confirmed the release candidate fixed it for them before it shipped. Source: https://github.com/huggingface/huggingface_hub/issues/4520
3. Take the question above and check if it matches what you're seeing. If it does, work through the verified fix step by step. Start with the cause described first, because that's what tells you the fix applies to your setup, then apply the changes in the order given. Verify by re-running whatever failed before, and expect the same behavior the thread author reported.

When to use

You are seeing this: The progress bars and the XET log just stop updating. Use this skill when you run into "hf download gets stuck on large downloads".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: hf-xet 1.5.2. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/huggingface/huggingface_hub/issues/4520

Someone asked: hf download on a large dataset always gets stuck after a while. The progress bars and the XET log just stop updating. Restarting lets it download a bit more, then it hangs again. Disabling XET did not change anything. The verified answer: This is a known bug in the XET download path that causes large downloads to hang. The Xet team identified it from logs and shipped a fix in hf-xet 1.5.2, so run pip install hf-xet==1.5.2 or newer and the stuck downloads should go through. A few people confirmed the release candidate fixed it for them before it shipped. Source: https://github.com/huggingface/huggingfacehub/issues/4520 Discussion and confirmation live at https://vectle.com/threads/thrWoO3NCXIpr3YVFdE0HNG_Q