TL;DR: Context: Ray Serve LLM troubleshooting FAQ: to use gated Hugging Face models, pass the access token through runtime_env env vars on the deployment, not baked into the image. [Ray Serve LLM troubleshooting FAQ]: for gated Hugging Face models, hand the token in through runtime_env env vars on the deployment, dont bake it into the image.

## Fix
1. Context: Ray Serve LLM troubleshooting FAQ: to use gated Hugging Face models, pass the access token through runtime_env env vars on the deployment, not baked into the image.
   Expected: this specific failure stops.
2. Use the get_deployment_options method on the LLMServer class to see the deployment options for each class.
   Expected: the request routes instead of erroring.

## Details
[Ray Serve LLM troubleshooting FAQ]: for gated Hugging Face models, hand the token in through runtime_env env vars on the deployment, dont bake it into the image. Build your LLMConfig with the model id and source, deployment autoscaling config, accelerator type, and runtime_env env_vars carrying HF_TOKEN. The LLMServer class get_deployment_options method shows you the options per deployment class. The pattern is runtime_env with env_vars carrying HF_TOKEN, plus the accelerator type you want, inside the LLMConfig.

## When to use
You hit exactly this: Anyscale: gated HF models need HF_TOKEN via runtime_env on the deployment (ray_serve_llm_auth) in Anyscale.

## When not to use
A different error, or the same symptom in a different tool. This page only covers the failure above.

## Compatibility
Anyscale.

---

## Vectle citations

- Exact version: <https://vectle.com/skills/skl_fP0gDPwqh5ak6fx3H6KN_A/versions/skv_lhLtVpRod94d-fi57EA5XA>
- Canonical HTML skill page: <https://vectle.com/skills/skl_fP0gDPwqh5ak6fx3H6KN_A>
- Current Markdown URL: <https://vectle.com/skills/skl_fP0gDPwqh5ak6fx3H6KN_A/markdown>

## Selected public attribution and original notices/references

```json
{
  "author": "issueatlas",
  "client": null,
  "notices_and_references": [
    "https://github.com/anyscale/ray/blob/HEAD/doc/source/serve/llm/troubleshooting.md"
  ]
}
```