Structured Output and Constrained Generation
Why parsing free text is the wrong integration point, how constrained decoding guarantees valid syntax, and what a guarantee about form does not give you about content.
An application that calls a model and then parses its prose is building on a foundation that fails a few percent of the time, unpredictably, in ways that are hard to handle. Getting structured output is the integration boundary that makes the rest of the system ordinary software, and there are three mechanisms with very different guarantees.
The three mechanisms
Prompting for a format asks the model to return JSON and hopes. It works most of the time, fails on edge cases, and produces the characteristic failures: a preamble before the JSON, a trailing explanation, a trailing comma, a missing brace, or a hallucinated field. It requires a retry loop and a parser tolerant of surrounding text.
Fine-tuning for a format raises the success rate substantially and does not guarantee anything, because a sampled token can still leave the grammar.
Constrained decoding guarantees validity by construction. At each step, compute which tokens could continue a valid string under the target grammar, mask the rest to negative infinity, and sample from what remains. The output cannot be invalid because invalid continuations were never available.
The implementation compiles the schema into a state machine or a grammar, tracks the parser state during generation, and derives the allowed token set at each position. Because vocabularies are large and the mask must be computed per token, efficient implementations precompute the token-to-state transitions, which is what makes the overhead small enough to be practical.
What the guarantee covers
Constrained decoding guarantees syntactic validity and nothing else. The JSON will parse and the fields will be present with the right types. Whether the values are correct, whether the enum choice is the right one, and whether a required string contains a real entity rather than a plausible invention are all outside what the constraint expresses.
This is the distinction that matters for system design: the constraint moves the failure from a parse error, which is loud and easy to handle, to a wrong value, which is silent. That is usually the right trade, and it is a trade rather than an improvement.
When it breaks
Constraining too early degrades reasoning. Forcing a model into a rigid structure from the first token prevents it from working through the problem, which measurably reduces quality on tasks that benefit from intermediate reasoning. The standard fix is to allow a free-form region for reasoning and constrain only the final answer.
Schema design affects accuracy, not just parsing. Field names carry semantic weight because the model conditions on them, so is_urgent and priority_flag elicit different behaviour. Ordering matters too, since fields generated earlier condition those after, so putting a reasoning field before the decision field improves the decision.
Deeply nested or highly permissive grammars are expensive. The state machine grows with the schema, and a grammar allowing many continuations at each point provides little constraint anyway. Flat schemas with tight enums are both cheaper and more effective.
Provider implementations differ. Structured output modes vary in whether they truly constrain decoding or retry until valid, and the difference matters for latency, for cost and for whether the guarantee actually holds. It is worth checking rather than assuming.
12 flashcards for this concept
Click a card to reveal the answer.