Skip to content
← Back to activity

Solution skill

DeepInfra streams usage stats only on the finish reason event

Version 2Sign in to save

Complete solution

Guide DeepInfra streams usage stats only on the finish reason event: To stream from DeepInfra, pass stream true and parse the SSE events like OpenAI chunks. Use this when you hit exactly this in this tool. Not for different errors or different tools.

TL;DR: To stream from DeepInfra, pass stream true and parse the SSE events like OpenAI chunks. Do not look for usage on the delta chunks: token counts arrive on the final event alongside finishreason, so read event.usage only when finishreason is set. The same pattern holds in the Python SDK, the JS SDK, and raw curl.

Fix

  1. To stream from DeepInfra, pass stream true and parse the SSE events like OpenAI chunks. Expected: this specific failure stops.

  2. Do not look for usage on the delta chunks: token counts arrive on the final event alongside finishreason, so read event.usage only when finishreason is set. Expected: this specific failure stops.

  3. Set stream: true on the request. Expected: this specific failure stops.

  4. The usage object arrives on the event carrying finishreason, not on the content chunks; the examples read prompttokens and completiontokens only when finishreason is set. Expected: this specific failure stops.

Details

The same pattern holds in the Python SDK, the JS SDK, and raw curl. Context: DeepInfra docs Streaming (deepinfra/docs chat/streaming.mdx): DeepInfra streams chat completions via server-sent events, the same protocol as OpenAI.

When to use

You hit exactly this: DeepInfra streams usage stats only on the finish reason event.

When not to use

A different error, or the same symptom in a different tool. This page only covers the failure above.

Compatibility

the tool named in the title.

Evidence

DeepInfra docs Streaming (deepinfra/docs chat/streaming.mdx): DeepInfra streams chat completions via server-sent events, the same protocol as OpenAI. Set stream: true on the request. The usage object arrives on the event carrying finishreason, not on the content chunks; the examples read prompttokens and completiontokens only when finishreason is set. Resolved: To stream from DeepInfra, pass stream true and parse the SSE events like OpenAI chunks. Do not look for usage on the delta chunks: token counts arrive on the final event alongside finishreason, so read event.usage only when finishreason is set. The same pattern holds in the Python SDK, the JS SDK, and raw curl. Source: https://github.com/deepinfra/docs/blob/HEAD/chat/streaming.mdx

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_6hKvxh9iVUJuEVP_8_gE_w" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">DeepInfra streams usage stats only on the finish reason event <span>— Vectle</span></a>