Structured Outputs (Constrained Decoding)

What are Structured Outputs?

Structured Outputs (also called Constrained Decoding or Guided Generation) is a set of techniques that force a large language model (LLM) to produce text matching a predefined format, structure, or schema. In practice, this usually means enforcing valid JSON output that matches a specific JSON Schema.

Developers used to rely on prompt engineering to coax models into outputting correctly formatted data. That approach was flaky: models would prepend text like 'Here is your JSON:', add trailing commas, or invent keys that didn't exist in the schema. Structured Outputs solve this at the inference engine level. The model's output is guaranteed to parse and comply with the schema, which makes it possible to plug LLMs into traditional software systems reliably.

By bounding the generative process, Structured Outputs let developers treat LLMs as deterministic functions or API endpoints rather than unpredictable conversational agents. The technique underlies agentic workflows, tool calling, and large-scale pipelines that convert unstructured text into structured data.

The mechanics of Constrained Decoding: context-free grammars (CFG) and logit masking

Structured Outputs achieve full schema compliance through Constrained Decoding, which intercepts the model's generation process at every token step and restricts its vocabulary based on the current state of a state machine or parser.

When a schema (like a JSON Schema) is provided, the inference engine compiles it into a Context-Free Grammar (CFG) or a Finite State Machine (FSM). During generation, before the model samples the next token, the engine consults this grammar.

1. Vocabulary evaluation: The model produces a probability distribution (logits) over its entire vocabulary for the next token.
2. Logit masking: The FSM determines which tokens are syntactically valid next steps according to the schema. For example, if the schema expects a boolean value, and the current output is "isActive": , the only valid next tokens are those representing true or false. The engine masks the logits for all invalid tokens by setting them to negative infinity.
3. Sampling: The model samples from the remaining, valid tokens.

This process guarantees that every generated token is a legal move within the grammar. Because the constraint applies at the level of token selection, the model cannot generate an invalid character, syntax error, or non-compliant key.

Comparison: prompt instruction vs. JSON mode vs. strict Structured Outputs

While JSON Mode guarantees the output is syntactically valid JSON (no missing quotes or brackets), it does not guarantee the JSON matches your specific schema. Strict Structured Outputs guarantee both: valid JSON that exactly matches the required keys, types, and constraints defined in your schema. This is one reason Structured Outputs are often paired with LLM guardrails, which validate model behavior beyond output formatting.

Provider API support (August 2026)

Every major provider now offers schema-guaranteed generation, though the parameter names and maturity differ:

• OpenAI shipped strict Structured Outputs in August 2024 and it remains the reference implementation: response_format with type: "json_schema" and strict: true in Chat Completions (the Responses API uses text.format). Strict mode enforces a subset of JSON Schema: every field must be listed in required, objects need additionalProperties: false, and some keywords (such as unbounded patternProperties) are unsupported. The same strict flag applies to function calling definitions.
• Anthropic long relied on a tool-based workaround (define a single tool whose input schema is your output schema, then force the model to call it). Native structured outputs launched in beta in November 2025 (output_format with the structured-outputs-2025-11-13 header) and are now generally available via output_config.format. A companion "strict tool use" option applies the same constrained decoding to tool-call parameters.
• Google Gemini accepts a responseSchema (with responseMimeType: "application/json"), and newer API versions accept standard JSON Schema directly via responseJsonSchema, closing an earlier gap where Gemini used its own OpenAPI-flavored schema subset.
• Open-weight serving engines expose the most flexible constraints. vLLM supports structured outputs through pluggable backends, with XGrammar as the common default and llguidance (Guidance) and Outlines as alternatives; SGLang ships XGrammar-based constrained decoding; llama.cpp uses GBNF grammars. These accept not just JSON Schema but arbitrary regular expressions, choice lists, and full context-free grammars.

Backend choice matters at scale: benchmarks published by SqueezeBits comparing vLLM backends found XGrammar's JIT-compiled, cached grammars fastest per output token when schemas are reused, while llguidance's per-token computation gives faster time-to-first-token on fresh schemas.

Schema definition and implementation example

In practice, defining structured outputs means defining a JSON Schema. Frameworks like Python's Pydantic are widely used for this because they let developers define schemas as native Python classes, which convert directly to JSON Schema for the LLM.

Here is an example of defining a schema for extracting user profiles using Pydantic, which could be passed to an LLM inference API supporting structured outputs:

from pydantic import BaseModel, Field
from typing import List, Optional

class Address(BaseModel):
    street: str = Field(description='The street address')
    city: str = Field(description='The city name')
    zip_code: str = Field(description='The postal code')

class UserProfile(BaseModel):
    name: str = Field(description='Full name of the user')
    age: int = Field(description='Age in years')
    is_active: bool = Field(description='Whether the user account is active')
    hobbies: List[str] = Field(description='List of user hobbies')
    address: Optional[Address] = Field(description='User address')

# The schema can be exported to JSON Schema to be sent to the LLM API
schema = UserProfile.model_json_schema()

When this schema is provided to a compatible API (like OpenAI's API with response_format={"type": "json_schema", "json_schema": {"schema": schema, "strict": True}}), the engine ensures the returned JSON strictly adheres to the UserProfile definition, never missing a required field or hallucinating an unknown one.

Performance, token latency, and grammar compilation overhead

Structured Outputs improve reliability, but they aren't free computationally. The overhead shows up in two places:

1. Grammar compilation (time to first token, TTFT): When a new, unique schema is submitted to an inference engine, it must be compiled into an FSM. This compilation step can add noticeable latency to the time to first token, especially for large or deeply nested schemas. Modern APIs cache compiled grammars, so if you reuse the same schema across many requests, you pay the compilation penalty only once.
2. Token generation overhead (tokens per second, TPS): During generation, the logit masking process requires evaluating the FSM state against the vocabulary at every step. Optimized inference engines, using frameworks like Outlines or Guidance, have reduced this overhead to near zero for most common grammars, but very complex, non-deterministic grammars can still slow generation slightly.

For production systems, the reliability gains outweigh the performance costs. The time saved by eliminating parsing errors, retry logic, and validation loops exceeds the milliseconds lost to logit masking.

Production use cases: tool calling, data extraction, and workflow orchestration

Structured Outputs changed how LLMs get deployed in production, shifting them from text generators to reasoning engines embedded inside software applications.

Tool calling (function calling)

When an agentic LLM decides to use a tool, such as querying a database or fetching weather data, it must provide arguments that match the tool's expected signature. This is the same mechanism behind protocols like Model Context Protocol, which standardizes how models discover and call tools. Structured Outputs ensure the model generates a precise JSON payload corresponding to the tool's parameters, preventing crashes caused by missing arguments or type mismatches.

Data extraction from unstructured text

Processing messy real-world data, such as parsing resumes into a database or extracting invoice details, requires rigid schema adherence. Structured Outputs let organizations transform large volumes of unstructured documents into clean, queryable SQL tables without human intervention.

Workflow orchestration and UI generation

Complex workflows often require the LLM to output a sequence of actions or a UI state representation. By enforcing a schema, the LLM generates JSON that drives front-end components directly or triggers downstream microservices, so the orchestration layer never receives malformed instructions.

Frequently Asked Questions

Does Constrained Decoding limit the model's creativity or reasoning? Yes and no. Restricting the vocabulary limits what the model can say, but that's usually the point when you're extracting data. If the model needs to reason before outputting data, a common pattern is to include a chain_of_thought string field at the top of the schema so it can think freely in text before committing to the structured fields. That field is unconstrained text, though, and it flows through to whatever consumes the schema's output. If the response is shown to end users, logged, or fed into another system, treat the chain_of_thought content as untrusted and potentially sensitive: it can leak internal reasoning, restate private context from the prompt, or (if the input was adversarial) echo injected instructions. Strip or review it before it leaves the pipeline rather than assuming it's safe because it sits inside a validated schema. Schema conformance is a syntax guarantee: it confirms the output has the right shape, not that its content is correct or safe.

Do structured outputs degrade output quality or reasoning? Research on this is mixed: some studies found strict format constraints measurably hurt reasoning-heavy tasks, others found negligible impact with well-designed schemas. Two mitigations have made the concern mostly historical. First, the chain_of_thought field pattern gives the model unconstrained space to reason inside the schema. Second, reasoning models generate their hidden thinking tokens before constrained decoding applies, so the deliberation is never masked; only the final answer is. Schema design still matters: descriptive field names and descriptions, sensible field ordering (reasoning fields before conclusion fields), and avoiding deeply nested optionality all improve accuracy.

Can I use Structured Outputs with open source models? Yes. Proprietary APIs popularized the feature, but open source inference servers like vLLM, TGI, and llama.cpp all support constrained decoding through frameworks like Outlines. You can enforce JSON schemas, regex patterns, and CFGs locally on open-weight models.

What happens if the model is forced into a corner? Because the FSM masks invalid tokens, there are rare cases where a model might have low probability for all valid tokens. In these situations, the model is forced to pick the 'least bad' valid option. This can sometimes lead to suboptimal or slightly hallucinated data within the valid constraints if the initial prompt was heavily contradictory to the expected schema.

Is JSON the only format supported? JSON is the industry standard for APIs, but Constrained Decoding works on grammars generally. You can constrain models to output valid XML, SQL queries, Python code, or strict regex formats, such as an output that exactly matches an email address pattern.