Structured Outputs and JSON Mode: Getting Reliable Data from LLMs

Every team that puts an LLM behind an API endpoint eventually hits the same wall: the model writes prose beautifully, but your downstream code needs a JSON object with exact field names and types. A single stray comma, a markdown code fence the model insists on wrapping around its answer, or a string where you expected an integer, and your parser throws. Multiply that by a few thousand requests a day and “mostly works” becomes an incident.

Structured outputs exist to close that gap. The pitch is simple: instead of asking nicely and hoping, you constrain the model so it is incapable of producing invalid output. Understanding how that constraint actually works, and where it still leaves gaps, is what separates a pipeline that survives production from one that pages someone at 2 a.m.

The Old Way: Prompting and Praying

The first generation of “get me JSON” techniques were prompt-engineering tricks: “Respond only with valid JSON, no other text”, a one-shot example, maybe a regex to strip a leading ```json fence. This works most of the time, which is exactly the problem. “Most of the time” means your error handling has to cover the long tail, and that tail includes trailing commentary, truncated output when the model hits a token limit mid-object, and occasional hallucinated fields that don’t exist in your schema.

None of this is the model being unreasonable. Language models generate tokens by sampling from a probability distribution over the entire vocabulary at each step. Nothing in that process inherently understands “this position must be a closing brace.” The model has learned, from training data, that JSON-looking text tends to be well-formed, but “tends to” is a statistical property, not a guarantee.

Constrained Decoding: How JSON Mode Actually Works

Real structured output support changes where the constraint lives: instead of hoping the model’s learned habits produce valid syntax, the decoding loop itself masks out any token that would violate the target grammar.

Concretely, the inference server maintains a state machine (or a pushdown automaton, since JSON nesting requires a stack) derived from your schema. At every generation step, before sampling, it computes which tokens are even legal given the current parse state, and sets the logits of every other token to negative infinity. If you’ve just opened {"name": " and the schema says name is a string, only tokens that continue a valid string, or close it, survive. The model cannot emit a stray brace no matter how confident it is in that token, because the sampler never gives it the chance.

This is a form of grammar-based sampling, and it’s a strictly stronger guarantee than a well-crafted prompt. The tradeoff is that it constrains syntax, not semantics. The model can still put a plausible-looking but wrong value in a valid slot; it just can’t produce a slot that violates the shape of your schema.

# Illustration of the masking idea (simplified, not a real inference server)
def constrained_step(logits, grammar_state):
    allowed_token_ids = grammar_state.allowed_next_tokens()
    mask = torch.full_like(logits, float("-inf"))
    mask[allowed_token_ids] = 0
    return logits + mask  # disallowed tokens become unsamplable

JSON Schema as the Contract

Most providers that support structured outputs accept a subset of JSON Schema to define the shape you want. Treat this schema as an API contract between you and the model, not as documentation.

{
  "type": "object",
  "properties": {
    "ticket_id": {"type": "string"},
    "priority": {"type": "string", "enum": ["low", "medium", "high", "urgent"]},
    "summary": {"type": "string", "maxLength": 200},
    "assignee": {"type": ["string", "null"]},
    "tags": {"type": "array", "items": {"type": "string"}}
  },
  "required": ["ticket_id", "priority", "summary", "assignee", "tags"],
  "additionalProperties": false
}

A few details here matter more than they look. additionalProperties: false isn’t decorative; without it, the model is free to bolt on extra fields, and some grammar implementations will happily generate them since nothing in the schema forbids it. "assignee": {"type": ["string", "null"]} explicitly allows null, which you need whenever a field is legitimately optional in your data model — see the nullable-fields pitfall below.

Calling It from Python

The exact request shape differs by provider, but the pattern is consistent enough that you can write a thin wrapper against a generic client interface and swap backends later.

from pydantic import BaseModel, Field
from typing import Literal, Optional

class TicketExtraction(BaseModel):
    ticket_id: str
    priority: Literal["low", "medium", "high", "urgent"]
    summary: str = Field(max_length=200)
    assignee: Optional[str]
    tags: list[str]

def extract_ticket(client, raw_email: str) -> TicketExtraction:
    response = client.chat(
        messages=[
            {"role": "system", "content": "Extract a support ticket from the email."},
            {"role": "user", "content": raw_email},
        ],
        response_schema=TicketExtraction.model_json_schema(),
        response_format="json_schema",
    )
    return TicketExtraction.model_validate_json(response.content)

Notice that the Pydantic model is the schema. That’s the pattern worth adopting: define your data model once, generate the JSON Schema from it, and validate the response back into the same model. You get static typing on both ends of the round trip for free.

Validation and Retry Loops

Constrained decoding guarantees syntactic validity, not correctness. The model can still return "priority": "urgent" for a password-reset request, or leave summary empty when your business logic requires content. This is where a validation-and-retry loop earns its keep.

def extract_with_retry(client, raw_email: str, max_attempts: int = 3) -> TicketExtraction:
    last_error = None
    for attempt in range(max_attempts):
        try:
            result = extract_ticket(client, raw_email)
            if not result.summary.strip():
                raise ValueError("summary must not be empty")
            return result
        except (ValidationError, ValueError) as e:
            last_error = e
            raw_email = (
                f"{raw_email}\n\n"
                f"[Previous attempt failed validation: {e}. Fix and retry.]"
            )
    raise RuntimeError(f"extraction failed after {max_attempts} attempts: {last_error}")

Feeding the validation error back into the next prompt, rather than just retrying blind, noticeably improves second-attempt success rates, because you’re giving the model a concrete, machine-generated reason rather than making it guess what went wrong.

Pitfalls Worth Knowing Before You Ship

Schema too strict. A schema with a dozen required fields and no defaults forces the model to invent a value even when the source text has no answer for that field. If a field is genuinely optional, mark it nullable and make your downstream code handle null explicitly, rather than pressuring the model into fabricating plausible-looking filler.

Enums as a correctness lever, not just a syntax lever. Constraining priority to a fixed enum doesn’t just guarantee valid JSON, it also eliminates an entire class of near-miss values like "High", "P1", or "critical" that a downstream switch statement would silently ignore. Whenever a field has a known, finite set of legal values, enumerate them in the schema instead of leaving it as a free-form string.

Nullable fields need type: ["string", "null"], not just "required": false. JSON Schema’s required array controls whether a key must be present; it says nothing about whether its value can be null. Removing a field from required without adding null to its type just means the model is unconstrained about whether to include the key at all, which reintroduces the exact ambiguity you were trying to remove.

Recursive or deeply nested schemas (a comment tree, a nested org chart) push some grammar compilers to their limits and can add noticeable latency to the first token, since the state machine has to be built before generation starts. If you hit this, flatten the structure or paginate: extract one level at a time instead of the whole tree in one shot.

Unicode and escaping in free-text fields. A summary field pulled from a support email with emoji, non-Latin scripts, or embedded quotes is still valid JSON once escaped, but it’s a common source of “the schema passed validation but the string looks mangled” bug reports. Test your extraction pipeline against real user-generated text, not just clean synthetic examples.

Comparing the Approaches

Approach	Syntax guarantee	Extra latency	Best for
Prompt-only (“respond in JSON”)	None	None	Prototypes, low-stakes internal tools
Post-hoc regex/repair	Partial	Low	Legacy pipelines you can’t touch upstream
JSON mode / structured outputs (grammar)	Full syntax	Small (grammar compile)	Production data extraction, agent tool calls
Tool/function calling with a schema	Full syntax, provider-managed	Small	Agent loops where the “output” is really an action

A Self-Hosted Note

If you’re running an open-weights model behind your own inference server (vLLM, TGI, llama.cpp with a grammar, and similar stacks all support this), constrained decoding is a local, free capability — you don’t need a hosted API to get it. It’s one of the more compelling reasons to self-host for extraction-heavy workloads: you get the same syntactic guarantee as a commercial structured-output endpoint, without sending the raw documents off-box.

Key Takeaways

• Structured output support works by masking invalid tokens at each decoding step, not by hoping the model behaves.
• This guarantees syntactic validity but not semantic correctness — you still need application-level validation.
• Define your schema once (a Pydantic model or equivalent) and derive both the request schema and the response parser from it.
• Use enums aggressively for any field with a known, finite set of values; it prevents an entire class of near-miss strings.
• Mark genuinely optional fields as nullable in the type, not just absent from required.
• Feed validation errors back into a retry prompt instead of retrying blind; it measurably improves success rates.
• Watch schema complexity on deeply nested or recursive structures — flatten or paginate if latency or reliability suffers.