Llama-3 models hallucinate when `stop` is not found in output

TL;DR

Issue togethercomputer/together-python#137 (closed, 3 comments): Llama-3 chat models on Together ignored the stop list: with stop=['test'] the model kept emitting repeated output ('bar foo foo fooassistant...') instead of stopping.

Steps

1. Issue togethercomputer/together-python#137 (closed, 3 comments): Llama-3 chat models on Together ignored the stop list: with stop=['test'] the model kept emitting repeated output ('bar foo foo fooassistant ...') instead of stopping. Llama-2 models did not have this problem.
2. Contributor orangetin explained the tokenizer ignores the end-of-turn token; adding '[|eot_id|]' to the stop list fixes it. Reporter Infrared1029 confirmed ('yeah works when i add it'). Contributor said they would also patch it server-side.

When to use

You are seeing this: Llama-2 models did not have this problem. Use this skill when you run into "Llama-3 models hallucinate when stop is not found in output".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Asked in gh:togethercomputer/together-python#137 (https://github.com/togethercomputer/together-python/issues/137): Llama-3 models hallucinate when stop is not found in output.. The issue report: Issue togethercomputer/together-python#137 (closed, 3 comments): Llama-3 chat models on Together ignored the stop list: with stop=['test'] the model kept emitting repeated output ('bar foo foo fooassistant ...') instead of stopping. Llama-2 models did not have this problem. The verified answer: Contributor orangetin explained the tokenizer ignores the end-of-turn token; adding '[|eotid|]' to the stop list fixes it. Reporter Infrared1029 confirmed ('yeah works when i add it'). Contributor said they would also patch it server-side. Lesson: always include [|eotid|] in stop sequences for Llama-3 models on Together.