Streaming not working on Azure OpenAI
Haystack: streaming not working on Azure OpenAI. The verified fix is Fix: Reporter jambudipa, after investigation with the maintainers: 'No changes to Haystack required, I was passing the stream handler in the wrong place.' The fix is to pass model_kwargs={'stream': True, 'stream_handler': custom_handler} on the PromptNode while constructing the PromptModel with the handler too - i.e. create the streaming PromptModel via create_streaming_*_prompt_model(custom_handler) rather than attaching the h
TL;DR
Fix: Reporter jambudipa, after investigation with the maintainers: 'No changes to Haystack required, I was passing the stream handler in the wrong place.' The fix is to pass modelkwargs={'stream': True, 'streamhandler': customhandler} on the PromptNode while constructing the PromptModel with the handler too - i.e. create the streaming PromptModel via createstreamingpromptmodel(customhandler) rather than attaching the handler only at the PromptNode level.
The error
Issue deepset-ai/haystack#5896 (closed, 12 comments): **Describe the bug** The `model_kwargs={"stream": True, "stream_handler": custom_handler}` on `PromptNode` has no effect when using a `PromptModel` based on an Azure OpenAI model. **Expected behavior** The answer output should stream as it does when using other prompt models. **Additional context** The following code works fine: ``` prompt_node = PromptNode( "gpt-3.5-turbo-16k", default_prompt_template=lfqa_prompt, api_key [your value], max_length= 3072, model_kwargs={"stream": True} ) ``` This streams as expected. **To Reproduce** Changing the prompt model like this: ``` # new prompt_azure_Fix
- Reporter jambudipa, after investigation with the maintainers: 'No changes to Haystack required, I was passing the stream handler in the wrong place.' The fix is to pass modelkwargs={'stream': True, 'streamhandler': customhandler} on the PromptNode while constructing the PromptModel with the handler too - i.e. create the streaming PromptModel via createstreamingpromptmodel(customhandler) rather than attaching the handler only at the PromptNode level.
Success check: the symptom above is gone and the original operation completes.
When to use
- You hit this exact symptom in haystack.
- You want the verified fix without digging through the thread.
When not to use
- A different error in haystack, even a similar-looking one.
- General haystack setup or credential questions.
Compatibility
- Haystack (deepset-ai/haystack)
Variant phrasings
- How to fix "streaming not working on Azure OpenAI" in haystack
- haystack: streaming not working on Azure OpenAI
Source
- https://github.com/deepset-ai/haystack/issues/5896