Immutable solution version
Llama 3 streaming repeats the previous request's first token
Complete solution
If your streamed Llama 3 responses start by echoing the first tokens of previous turns, you are on a client version with the streaming bug, not a model problem. The maintainer reproduced it in the client alone after ruling out stop sequences, and the fix is in replicate-python 0.25.2. Upgrade past 0.25.2 and the repeated prefix tokens go away.
Llama 3 streaming repeats the previous request's first token
Streaming meta/meta-llama-3-70b-instruct produced a strange artifact: each new response in a chat started by repeating earlier turns' first tokens, first "Hi", then "I", then "How", growing with the conversation. Maintainer mattt first suspected the model's stop sequences and pushed a new model build, then reproduced the issue in the client in isolation. The client-side fix shipped in replicate-python 0.25.2.
How to handle it
If your streamed Llama 3 responses start by echoing the first tokens of previous turns, you are on a client version with the streaming bug, not a model problem. The maintainer reproduced it in the client alone after ruling out stop sequences, and the fix is in replicate-python 0.25.2. Upgrade past 0.25.2 and the repeated prefix tokens go away. If you still see it, check the dashboard predictions to confirm whether the repetition is in the model output or the client rendering. Source: https://github.com/replicate/replicate-python/issues/287
Evidence
Question asked in the thread: Streaming meta/meta-llama-3-70b-instruct produced a strange artifact: each new response in a chat started by repeating earlier turns' first tokens, first "Hi", then "I", then "How", growing with the conversation. Maintainer mattt first suspected the model's stop sequences and pushed a new model build, then reproduced the issue in the client in isolation. The client-side fix shipped in replicate-python 0.25.2.
Verified answer: If your streamed Llama 3 responses start by echoing the first tokens of previous turns, you are on a client version with the streaming bug, not a model problem. The maintainer reproduced it in the client alone after ruling out stop sequences, and the fix is in replicate-python 0.25.2. Upgrade past 0.25.2 and the repeated prefix tokens go away. If you still see it, check the dashboard predictions to confirm whether the repetition is in the model output or the client rendering. Source: https://github.com/replicate/replicate-python/issues/287
Thread: https://vectle.com/threads/thrfGDqRWh0CuLeam-ZeAdMw
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_MiWzsZmPYG2cv45o8b9CjQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Llama 3 streaming repeats the previous request's first token <span>— Vectle</span></a>