Anyscale: gated HF models need HF_TOKEN via runtime_env on the deployment (ray_serve_llm_auth)

Anyscale: gated HF models need HFTOKEN via runtimeenv on the deployment (rayservellm_auth)

TL;DR

[Ray Serve LLM troubleshooting FAQ]: for gated Hugging Face models, hand the token in through runtimeenv env vars on the deployment, dont bake it into the image. Build your LLMConfig with the model id and source, deployment autoscaling config, accelerator type, and runtimeenv envvars carrying HFTOKEN. The LLMServer class getdeploymentoptions method shows you the options per deployment class.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

Context: Ray Serve LLM troubleshooting FAQ: to use gated Hugging Face models, pass the access token through runtimeenv env vars on the deployment, not baked into the image. Use the getdeploymentoptions method on the LLMServer class to see the deployment options for each class. The pattern is runtimeenv with envvars carrying HFTOKEN, plus the accelerator type you want, inside the LLMConfig.

Source: Ray Serve LLM troubleshooting FAQ: to use gated Hugging Face models, pass the access token through runtimeenv env vars on the deployment, not baked into the image. Use the getdeploymentoptions method on the LLMServer class to see the deployment options for each class. The pattern is runtimeenv with envvars carrying HFTOKEN, plus the accelerator type you want, inside the LLMConfig.

Verified resolution: [Ray Serve LLM troubleshooting FAQ]: for gated Hugging Face models, hand the token in through runtimeenv env vars on the deployment, dont bake it into the image. Build your LLMConfig with the model id and source, deployment autoscaling config, accelerator type, and runtimeenv envvars carrying HFTOKEN. The LLMServer class getdeploymentoptions method shows you the options per deployment class.

Source URL: https://github.com/anyscale/ray/blob/HEAD/doc/source/serve/llm/troubleshooting.md