Stopping Criteria and EOS Calibration

A production summariser starts returning answers that stop mid-word. The model did not change and the prompt did not change; the logs show finish_reason: length. Somebody raised the maximum input size, the prompts got longer, and an output budget that was generous at 400 tokens now truncates a third of responses. The system did not fail loudly. It shipped incomplete text with a 200 status code.

Termination is the least examined part of the decoding stack and one of the more common sources of silent quality loss.

Three independent mechanisms

The EOS token is a real vocabulary entry with a real probability at every step. The model learned when to emit it from training data, so it is calibrated on the length distribution of that data and no other. Ask a model trained on paragraph-length answers for a one-line answer and it will often continue past the point the answer is complete, because a short response is out of distribution.

Stop sequences are string matches applied to the detokenised output. They live outside the model and are checked by the serving layer. Their characteristic bug is tokenisation: a newline pair may be a single token in one tokeniser and two in another, and a stop sequence that does not align to a token boundary can be missed entirely or fire one character late, leaving a fragment.

The token budget is a hard cap. When it fires, the model was mid-sentence and had no intention of stopping. Any pipeline that treats a budget stop the same as an EOS stop will parse truncated JSON, average truncated summaries into an evaluation, and store half a sentence as a final answer. The finish_reason field exists precisely to distinguish these, and it is routinely ignored.

Why EOS is systematically mis-calibrated

Under maximum likelihood training a sequence model assigns probability to a whole string, and each additional token multiplies in a factor below one. Length is therefore penalised by construction. Stahlberg and Byrne quantified how far this goes: with exact search over a trained translation model, more than half of WMT15 English-German sentences had the empty string as the global optimum (Stahlberg & Byrne, EMNLP 2019, arXiv:1908.10090). Beam search's failure to find that optimum is load-bearing.

Post-training pushes hard in the opposite direction. Human preference data favours longer, more thorough answers, so reward models learn a length bias and RLHF amplifies it. Singhal and colleagues found that reward improvements during RLHF were largely driven by increasing response length, that a purely length-based reward reproduced most of the downstream gains, and that the reward model was the dominant source of the bias (Singhal et al., COLM 2024, arXiv:2310.03716).

A modern instruction-tuned model therefore carries two opposing pressures on EOS: a pretraining prior that favours stopping early and a preference-tuning gradient that favours continuing. Neither is calibrated to any particular application's desired length.

Interaction with constrained decoding

Grammar-constrained and schema-constrained decoding mask the logits to tokens a grammar allows. If the grammar is in a state where a JSON object is not yet closed, EOS is masked out, which is correct: the model must not stop mid-object. The hazard is a masked EOS combined with a token budget, because a model that wanders inside a long array will exhaust the budget and return syntactically invalid output despite the constraint machinery.

Format restriction also has a measured capability cost. Tam and colleagues found significant declines in reasoning performance under format restrictions, with stricter constraints producing greater degradation (Tam et al., 2024, arXiv:2408.02442). Forcing structure and forcing termination are both constraints on the same distribution, and both trade quality for guarantees.

When it breaks

Chat templates and EOS drift. Instruction-tuned models often use a distinct end-of-turn token rather than the base model's EOS. Fine-tune on data that omits it and the model never learns to stop; serve with a template whose end-of-turn token is missing from the stop list and generation runs to the budget every time. This is the most common cause of a fine-tune that will not stop talking.

Budget stops corrupt evaluations. A benchmark harness that truncates 5% of generations and scores them as ordinary outputs reports a number that is partly a measurement of the max_tokens setting.

Streaming hides the fragment. With server-sent events the client has already rendered the text when truncation occurs, so the failure appears as a formatting oddity rather than an error.

Reasoning models turn the budget into a capability knob. When trace and answer share one budget, a low cap does not truncate the answer, it truncates the thinking that would have produced it. The observable is a wrong answer rather than a truncated one, which makes the cause considerably harder to find.