Skip to content
← Back to activity

Solution skill

Streaming not working on Azure OpenAI

Version 2Sign in to save

Complete solution

Haystack: streaming not working on Azure OpenAI. The verified fix is Fix: Reporter jambudipa, after investigation with the maintainers: 'No changes to Haystack required, I was passing the stream handler in the wrong place.' The fix is to pass model_kwargs={'stream': True, 'stream_handler': custom_handler} on the PromptNode while constructing the PromptModel with the handler too - i.e. create the streaming PromptModel via create_streaming_*_prompt_model(custom_handler) rather than attaching the h

TL;DR

Fix: Reporter jambudipa, after investigation with the maintainers: 'No changes to Haystack required, I was passing the stream handler in the wrong place.' The fix is to pass modelkwargs={'stream': True, 'streamhandler': customhandler} on the PromptNode while constructing the PromptModel with the handler too - i.e. create the streaming PromptModel via createstreamingpromptmodel(customhandler) rather than attaching the handler only at the PromptNode level.

The error

Issue deepset-ai/haystack#5896 (closed, 12 comments): **Describe the bug** The `model_kwargs={"stream": True, "stream_handler": custom_handler}` on `PromptNode` has no effect when using a `PromptModel` based on an Azure OpenAI model. **Expected behavior** The answer output should stream as it does when using other prompt models. **Additional context** The following code works fine: ``` prompt_node = PromptNode( "gpt-3.5-turbo-16k", default_prompt_template=lfqa_prompt, api_key [your value], max_length= 3072, model_kwargs={"stream": True} ) ``` This streams as expected. **To Reproduce** Changing the prompt model like this: ``` # new prompt_azure_

Fix

  1. Reporter jambudipa, after investigation with the maintainers: 'No changes to Haystack required, I was passing the stream handler in the wrong place.' The fix is to pass modelkwargs={'stream': True, 'streamhandler': customhandler} on the PromptNode while constructing the PromptModel with the handler too - i.e. create the streaming PromptModel via createstreamingpromptmodel(customhandler) rather than attaching the handler only at the PromptNode level.

Success check: the symptom above is gone and the original operation completes.

When to use

  • You hit this exact symptom in haystack.
  • You want the verified fix without digging through the thread.

When not to use

  • A different error in haystack, even a similar-looking one.
  • General haystack setup or credential questions.

Compatibility

  • Haystack (deepset-ai/haystack)

Variant phrasings

  • How to fix "streaming not working on Azure OpenAI" in haystack
  • haystack: streaming not working on Azure OpenAI

Source

  • https://github.com/deepset-ai/haystack/issues/5896

Evidence

In gh:deepset-ai/haystack#5896 (https://github.com/deepset-ai/haystack/issues/5896), someone reported: Issue deepset-ai/haystack#5896 (closed, 12 comments): Describe the bug The model_kwargs={"stream": True, "stream_handler": custom_handler} on PromptNode has no effect when using a PromptModel based on an Azure OpenAI model. Expected behavior The answer output should stream as it does when using other prompt models. Additional context The following code works fine: `` prompt_node = PromptNode( "gpt-3.5-turbo-16k", default_prompt_template=lfqa_prompt, api_key [your value], max_length= 3072, model_kwargs={"stream": True} ) This streams as expected. **To Reproduce** Changing the prompt model like this: `` # new promptazure The verified resolution from the thread: Reporter jambudipa, after investigation with the maintainers: 'No changes to Haystack required, I was passing the stream handler in the wrong place.' The fix is to pass modelkwargs={'stream': True, 'streamhandler': customhandler} on the PromptNode while constructing the PromptModel with the handler too - i.e. create the streaming PromptModel via createstreamingpromptmodel(customhandler) rather than attaching the handler only at the PromptNode level. Contributor mark-norgate-ons confirmed it was just a config issue.

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_OtGWjt-Ljw8Ozvm8FLHNkA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Streaming not working on Azure OpenAI <span>— Vectle</span></a>