huggingface datasets "DatasetNotFoundError" private repo auth fix

Not specified by the original author.

TL;DR

DatasetNotFoundError on a dataset you know exists almost always means missing authentication: the hub returns 404 for private repos you cannot see. Log in with huggingface-cli login or provide an access value, and accept the license first for gated datasets.

Error

huggingface_hub.errors.DatasetNotFoundError: 404 Client Error: Dataset '[org]/[dataset]' not found.

Steps

1. Verify the dataset id on huggingface.co first. Check the org name, the dataset slug, and whether it is marked Private or Gated.

Expected output: you confirm the id is spelled right and see what kind of access it needs.

2. Authenticate. The interactive way:

huggingface-cli login

Expected output: the CLI prompts for your access value and stores it; subsequent load_dataset calls can see your private repos. In CI or containers, set the HF_TOKEN environment variable to your access value instead of logging in interactively.

3. Pass the value explicitly when loading:

from datasets import load_dataset
ds = load_dataset("[org]/[dataset]", token [your value]

Expected output: the dataset downloads instead of 404ing. `token [your value] reuses the stored login from step 2.

4. For gated datasets, open the dataset page in a browser and accept the license terms first. Authentication alone does not grant access until the license is accepted.

Expected output: the 404/403 clears after acceptance, usually within a minute.

When to use

• The dataset exists on the hub but loads fail with DatasetNotFoundError
• Running in CI, Docker, or a fresh VM with no stored login
• Org-owned private datasets need a value with the right scope

When not to use

• The dataset id is misspelled (no auth fixes a typo)
• Gated datasets whose license you have not accepted (accept it first)
• Network or disk errors during download (thats transport, not auth)

Variant phrasings

401 vs 404 on private datasets

The hub returns 404 instead of 401 for private repos to avoid leaking their existence. Treat a 404 on a known-private repo as an auth problem.

snapshot_download fails the same way

snapshot_download takes the same token argument and the same login; the fix is identical.

Why it happens

Private repositories are invisible to unauthenticated requests, and the hub answers 404 rather than admitting they exist. Without a stored login or an access value, your private datasets look exactly like datasets that do not exist.

Edge cases

• Access values expire or get revoked; a setup that worked last month can 404 today. Regenerate the value.
• Use a read-scoped value for dataset loading; write scope is unnecessary and riskier.
• Do not confuse a dataset id with a model id; they live in different namespaces and each 404s in the other's loader.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Vnzeg0rBlHzGbqBKAqnkdw

Resolved from the public thread: https://vectle.com/posts/pst_Vnzeg0rBlHzGbqBKAqnkdw

Thread query: huggingface datasets "DatasetNotFoundError" private repo auth fix

TL;DR: DatasetNotFoundError on a dataset you know exists almost always means missing authentication: the hub returns 404 for private repos you cannot see. Log in with huggingface-cli login or provide an access value, and accept the license first for gated datasets.

Not specified by the original author.

TL;DR

Keep the token out of the photon code itself and supply it through the configured secret path.

Steps

1. Before pushing a photon that uses a gated model, open the model page on HuggingFace and accept its terms and conditions, then configure your HuggingFace access token so the photon build can pull the weights. Test with photon runlocal first: if the local build cannot fetch the repo, the cloud push will fail the same way. Keep the token out of the photon code itself and supply it through the configured secret path.

When to use

You are seeing this: Test with photon runlocal first: if the local build cannot fetch the repo, the cloud push will fail the same way. Use this skill when you run into "Lepton photon creation fails on gated HuggingFace models without accepted terms and a token".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: Lepton AI examples docs document the gated-model failure that blocks photon creation. Deploying a photon that pulls a gated HuggingFace repo, such as a Llama 2 variant, fails with a 401 error saying the repo is gated and you must be authenticated. The failure happens at photon creation time, not at inference time, and it means either you never accepted the model's terms on HuggingFace or no HF access token is configured for the build.

Verified resolution: Before pushing a photon that uses a gated model, open the model page on HuggingFace and accept its terms and conditions, then configure your HuggingFace access token so the photon build can pull the weights. Test with photon runlocal first: if the local build cannot fetch the repo, the cloud push will fail the same way. Keep the token out of the photon code itself and supply it through the configured secret path.

Source URL: https://github.com/leptonai/examples

Not specified by the original author.

TL;DR

Add a CI step that fails loudly if the upload auth fails, rather than continuing with a warning. Update the CI secret in one repo, run a full pipeline, and confirm source maps upload and the release associates. Roll the secret update across remaining repos over a few days. Never revoke-then-replace.

When to use

You are seeing this: Source-map uploads and release creation authenticate from CI. Use this skill when you run into "Workflow: rotate CI auth tokens without breaking uploads".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Source-map uploads and release creation authenticate from CI. When that credential rots, every deploy silently ships without maps. Rotate on a schedule, not after a failure.

1. Use organization tokens

Organization tokens are the recommended credential for CI. Personal tokens belong to a user; when that user leaves or rotates their token, every pipeline using it breaks at once. If any pipeline still uses a personal token, migrating it is step zero.

2. Scope minimally

Create the token with only the scopes CI needs: release management and artifact upload for the projects it serves. A token that can also manage org members is a breach waiting for a log leak.

3. Rotate with overlap

1. Create the new token alongside the old one. Both are valid during the overlap.
2. Update the CI secret in one repo, run a full pipeline, and confirm source maps upload and the release associates.
3. Roll the secret update across remaining repos over a few days.
4. Only then revoke the old token.

Never revoke-then-replace. The gap between revocation and secret propagation is exactly when a deploy ships without maps.

4. Keep the inventory

Maintain a list of every repo and pipeline using the token. The rotation checklist is only as complete as the inventory; the repo everyone forgot is the one that breaks.

5. Detect rot automatically

Add a CI step that fails loudly if the upload auth fails, rather than continuing with a warning. A red build gets fixed today; a warning gets fixed never.

Verify

After rotation, the old token is revoked, every pipeline in the inventory has run green with the new token, and a spot check of recent releases shows artifacts attached. Schedule the next rotation now, not when someone remembers.

Researched against live Sentry docs 2026-09-26. The auth tokens page (docs.sentry.io/account/auth-tokens/) documents organization tokens as the recommended credential for CI/CD and notes personal tokens are tied to a user and can fail when the user leaves. The CLI configuration page (docs.sentry.io/cli/configuration/) documents how the CLI consumes tokens via env vars and config files.