My LLM Can't Stop Generating, How to Fix It?
Detect and fix issues with the EOS token
After fine-tuning a large language model (LLM) on your data, you might observe that the model doesn’t know when to stop generating tokens. Although the first tokens answer your prompt appropriately, the model continues to produce irrelevant tokens until it reaches the maximum sequence length.
This is a very common issue.
This issue arises when the end-of-sequence (EOS) token is not properly configured. This can occur with models like Llama 3, Qwen2, and many other LLMs.
Properly configuring the EOS token for fine-tuning can be challenging and may require several iterations to find an effective solution.
In this article, I present and implement three simple tests to diagnose issues with the EOS token. Using Llama 3 as an example, we will see how to address each issue effectively. These tests and solutions are applicable to any generative LLM, ensuring you can resolve similar problems regardless of the model you're working with.
The following notebook implements the tests and shows how to teach LLMs when to stop generating:
What Is the EOS Token?
The EOS token is a special token that triggers the end of the generation.
Stop reason: eos token found
It means the model ended generation because it produced the special end-of-sequence token (EOS). In practice, this is the “normal” completion path: the model decided it was done and emitted EOS, so the runtime stopped decoding. This is different from stopping due to a max token limit, a user-defined stop string, or an interruption/cancellation.
Most inference frameworks won’t tell you explicitly why they stop generating. It’s either because they reach the maximum sequence length defined by the user or the framework, or because the model sent the signal to stop generating, i.e., it outputs the EOS token.
Generative (auto-regressive) LLMs generate the tokens one by one. If the LLM generates the EOS token, the inference framework will stop the generation.
Most LLMs have an EOS special token in their vocabulary. Some examples of EOS tokens: