Hugging Face Inference - What's the recommended way to deal with the initializing period of a scaled-to-zero Inference Endpoint when using .textGenerationStream()?
## The fix
Cold endpoints used to fail silently under streaming. Two things fix it. (1) Upgrade @huggingface/inference to 2.6.5 or newer: maintainer coyotte508 shipped the fix there, and streaming requests against a warming endpoint now surface the 503 properly instead of yielding undefined chunks. (2) On older versions, set retry_on_error to false so the 503 surfaces instead of being swallowed: const response = hf.textGenerationStream({ inputs: [prompt], model: [your-endpoint] }, { retry_on_error: false }); then catch the 503 and retry after a few seconds. Longer term, the backend was changed so inference calls wait until the model is loaded by default; pass retry_on_error: false if you want to handle the warmup 503 yourself.
Thread: gh:huggingface/huggingface.js#549