Immutable solution version
Llama-3 models hallucinate when `stop` is not found in output.
Complete solution
How to fix 'Llama-3 models hallucinate when `stop` is not found in output.' in Together AI, straight from the verified resolution in gh:togethercomputer/together-python#137. Root cause plus what to change.
The problem
Issue togethercomputer/together-python#137 (closed, 3 comments): Llama-3 chat models on Together ignored the stop list: with stop=['test'] the model kept emitting repeated output ('bar foo foo fooassistant ...') instead of stopping. Llama-2 models did not have this problem.
The verified resolution
Contributor orangetin explained the tokenizer ignores the end-of-turn token; adding '<|eot_id|>' to the stop list fixes it. Reporter Infrared1029 confirmed ('yeah works when i add it'). Contributor said they would also patch it server-side.
Rule of thumb
always include <|eot_id|> in stop sequences for Llama-3 models on Together.
Evidence
Asked in gh:togethercomputer/together-python#137 (https://github.com/togethercomputer/together-python/issues/137): Llama-3 models hallucinate when stop is not found in output.. The issue report: Issue togethercomputer/together-python#137 (closed, 3 comments): Llama-3 chat models on Together ignored the stop list: with stop=['test'] the model kept emitting repeated output ('bar foo foo fooassistant ...') instead of stopping. Llama-2 models did not have this problem. The verified answer: Contributor orangetin explained the tokenizer ignores the end-of-turn token; adding '<|eotid|>' to the stop list fixes it. Reporter Infrared1029 confirmed ('yeah works when i add it'). Contributor said they would also patch it server-side. Lesson: always include <|eotid|> in stop sequences for Llama-3 models on Together.
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_W9_wq3WJPey6c4HI0Zm-2g" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Llama-3 models hallucinate when `stop` is not found in output. <span>— Vectle</span></a>