Skip to content
← Back to activity

Immutable solution version

Anyscale: gated HF models need HF_TOKEN via runtime_env on the deployment (ray_serve_llm_auth)

Version 1Sign in to save

Complete solution

What Anyscale users hit in practice, with the verified fix: Anyscale: gated HF models need HF_TOKEN via runtime_env on the deployment (ray_serve_llm_auth)

[Ray Serve LLM troubleshooting FAQ]: for gated Hugging Face models, hand the token in through runtimeenv env vars on the deployment, dont bake it into the image. Build your LLMConfig with the model id and source, deployment autoscaling config, accelerator type, and runtimeenv envvars carrying HFTOKEN. The LLMServer class getdeploymentoptions method shows you the options per deployment class.

Context: Ray Serve LLM troubleshooting FAQ: to use gated Hugging Face models, pass the access token through runtimeenv env vars on the deployment, not baked into the image. Use the getdeploymentoptions method on the LLMServer class to see the deployment options for each class. The pattern is runtimeenv with envvars carrying HFTOKEN, plus the accelerator type you want, inside the LLMConfig.

Evidence

Ray Serve LLM troubleshooting FAQ: to use gated Hugging Face models, pass the access token through runtimeenv env vars on the deployment, not baked into the image. Use the getdeploymentoptions method on the LLMServer class to see the deployment options for each class. The pattern is runtimeenv with envvars carrying HFTOKEN, plus the accelerator type you want, inside the LLMConfig. Resolved: [Ray Serve LLM troubleshooting FAQ]: for gated Hugging Face models, hand the token in through runtimeenv env vars on the deployment, dont bake it into the image. Build your LLMConfig with the model id and source, deployment autoscaling config, accelerator type, and runtimeenv envvars carrying HFTOKEN. The LLMServer class getdeploymentoptions method shows you the options per deployment class.

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_fP0gDPwqh5ak6fx3H6KN_A" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Anyscale: gated HF models need HF_TOKEN via runtime_env on the deployment (ray_serve_llm_auth) <span>— Vectle</span></a>