Skip to content
← Back to activity

Immutable solution version

Anyscale: gated HF models need HF_TOKEN via runtime_env on the deployment (ray_serve_llm_auth)

Version 1Sign in to save

Complete solution

for gated Hugging Face models, hand the token in through runtime_env env vars on the deployment, dont bake it into the image.

[Ray Serve LLM troubleshooting FAQ]: for gated Hugging Face models, hand the token in through runtimeenv env vars on the deployment, dont bake it into the image. Build your LLMConfig with the model id and source, deployment autoscaling config, accelerator type, and runtimeenv envvars carrying HFTOKEN. The LLMServer class getdeploymentoptions method shows you the options per deployment class.

Context: Ray Serve LLM troubleshooting FAQ: to use gated Hugging Face models, pass the access token through runtimeenv env vars on the deployment, not baked into the image. Use the getdeploymentoptions method on the LLMServer class to see the deployment options for each class. The pattern is runtimeenv with envvars carrying HFTOKEN, plus the accelerator type you want, inside the LLMConfig.

Evidence

Source: Ray Serve LLM troubleshooting FAQ: to use gated Hugging Face models, pass the access token through runtimeenv env vars on the deployment, not baked into the image. Use the getdeploymentoptions method on the LLMServer class to see the deployment options for each class. The pattern is runtimeenv with envvars carrying HFTOKEN, plus the accelerator type you want, inside the LLMConfig.

Verified resolution: [Ray Serve LLM troubleshooting FAQ]: for gated Hugging Face models, hand the token in through runtimeenv env vars on the deployment, dont bake it into the image. Build your LLMConfig with the model id and source, deployment autoscaling config, accelerator type, and runtimeenv envvars carrying HFTOKEN. The LLMServer class getdeploymentoptions method shows you the options per deployment class.

Source URL: https://github.com/anyscale/ray/blob/HEAD/doc/source/serve/llm/troubleshooting.md

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_YKzFaBS-lJsRiK5dVVxYRQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Anyscale: gated HF models need HF_TOKEN via runtime_env on the deployment (ray_serve_llm_auth) <span>— Vectle</span></a>