Anyscale: gated HF models need HF_TOKEN via runtime_env on the deployment (ray_serve_llm_auth)

TL;DR: Context: Ray Serve LLM troubleshooting FAQ: to use gated Hugging Face models, pass the access token through runtimeenv env vars on the deployment, not baked into the image. [Ray Serve LLM troubleshooting FAQ]: for gated Hugging Face models, hand the token in through runtimeenv env vars on the deployment, dont bake it into the image.

Fix

1. Context: Ray Serve LLM troubleshooting FAQ: to use gated Hugging Face models, pass the access token through runtime_env env vars on the deployment, not baked into the image. Expected: this specific failure stops.

2. Use the getdeploymentoptions method on the LLMServer class to see the deployment options for each class. Expected: the request routes instead of erroring.

Details

[Ray Serve LLM troubleshooting FAQ]: for gated Hugging Face models, hand the token in through runtimeenv env vars on the deployment, dont bake it into the image. Build your LLMConfig with the model id and source, deployment autoscaling config, accelerator type, and runtimeenv envvars carrying HFTOKEN. The LLMServer class getdeploymentoptions method shows you the options per deployment class. The pattern is runtimeenv with envvars carrying HF_TOKEN, plus the accelerator type you want, inside the LLMConfig.

When to use

You hit exactly this: Anyscale: gated HF models need HFTOKEN via runtimeenv on the deployment (rayservellm_auth) in Anyscale.

When not to use

A different error, or the same symptom in a different tool. This page only covers the failure above.

Compatibility

Anyscale.

Ray Serve LLM troubleshooting FAQ: to use gated Hugging Face models, pass the access token through runtimeenv env vars on the deployment, not baked into the image. Use the getdeploymentoptions method on the LLMServer class to see the deployment options for each class. The pattern is runtimeenv with envvars carrying HFTOKEN, plus the accelerator type you want, inside the LLMConfig. Resolved: [Ray Serve LLM troubleshooting FAQ]: for gated Hugging Face models, hand the token in through runtimeenv env vars on the deployment, dont bake it into the image. Build your LLMConfig with the model id and source, deployment autoscaling config, accelerator type, and runtimeenv envvars carrying HFTOKEN. The LLMServer class getdeploymentoptions method shows you the options per deployment class.