Skip to content
← Back to activity

Immutable solution version

Llama-3 models hallucinate when `stop` is not found in output.

Version 1Sign in to save

Complete solution

How to fix 'Llama-3 models hallucinate when `stop` is not found in output.' in Together AI, straight from the verified resolution in gh:togethercomputer/together-python#137. Root cause plus what to change.

The problem

Issue togethercomputer/together-python#137 (closed, 3 comments): Llama-3 chat models on Together ignored the stop list: with stop=['test'] the model kept emitting repeated output ('bar foo foo fooassistant ...') instead of stopping. Llama-2 models did not have this problem.

The verified resolution

Contributor orangetin explained the tokenizer ignores the end-of-turn token; adding '<|eot_id|>' to the stop list fixes it. Reporter Infrared1029 confirmed ('yeah works when i add it'). Contributor said they would also patch it server-side.

Rule of thumb

always include <|eot_id|> in stop sequences for Llama-3 models on Together.

Evidence

Asked in gh:togethercomputer/together-python#137 (https://github.com/togethercomputer/together-python/issues/137): Llama-3 models hallucinate when stop is not found in output.. The issue report: Issue togethercomputer/together-python#137 (closed, 3 comments): Llama-3 chat models on Together ignored the stop list: with stop=['test'] the model kept emitting repeated output ('bar foo foo fooassistant ...') instead of stopping. Llama-2 models did not have this problem. The verified answer: Contributor orangetin explained the tokenizer ignores the end-of-turn token; adding '<|eotid|>' to the stop list fixes it. Reporter Infrared1029 confirmed ('yeah works when i add it'). Contributor said they would also patch it server-side. Lesson: always include <|eotid|> in stop sequences for Llama-3 models on Together.

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_W9_wq3WJPey6c4HI0Zm-2g" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Llama-3 models hallucinate when `stop` is not found in output. <span>— Vectle</span></a>