A fintech company built an AI-powered invoice processor. It worked great in testing. In production, 40% of requests failed. The model returned JSON with inconsistent field names, missing required properties, and markdown formatting wrapped around the payload. The engineering team spent more time writing parsing workarounds than building features.

This is the most common failure mode in production AI systems, and it's entirely preventable.

The Three Approaches

There are three ways to get structured output from an LLM: prompt-only formatting, API-level JSON mode, and constrained decoding. Each has real tradeoffs that most guides gloss over.

Prompt-only means you describe the schema in your prompt and hope the model complies. For simple flat objects, this works surprisingly well. A clear schema description, one example, and temperature=0 gets you above 95% compliance on basic JSON. No special API flags needed, works with any model, and you keep full reasoning quality.

JSON mode is a provider-level flag that guarantees syntactically valid JSON. OpenAI's response_format: { type: "json_object" }, Gemini's response_mime_type: "application/json". The output will parse, but it won't necessarily match your schema. You get valid JSON with the wrong fields.

Constrained decoding is the heavy machinery. The provider compiles your JSON schema into a grammar, then masks invalid tokens during generation so the model literally cannot produce non-conforming output. As of late 2025, all three major providers offer this: Claude's json_schema parameter, OpenAI's Structured Outputs, and Gemini's response_schema. Under the hood, engines like Microsoft's llguidance process token masks in roughly 50 microseconds per token for a 128K tokenizer.

The Part Nobody Tells You: Structure Can Make Your Model Dumber

Here's the counterintuitive finding. Researchers at EMNLP 2024 tested LLMs on reasoning benchmarks with and without JSON formatting constraints. Strict JSON mode significantly degraded performance on tasks like math reasoning (GSM8K) and multi-step logic problems.

Why? Constrained decoding doesn't change how the model thinks. The model computes token probabilities normally, then invalid tokens get masked after the fact. When the model's preferred next token violates the schema, it's forced to pick a lower-probability alternative. On reasoning tasks, this introduces noise at exactly the wrong moments. Worse, the model sometimes places the answer field before the reasoning field in the JSON, short-circuiting its own chain of thought.

The same research found the opposite effect on classification tasks. Structure reduced ambiguity and improved accuracy. The pattern is clear: the more reasoning a task requires, the more structure hurts.

A 2025 follow-up called CRANE confirmed this and proposed a fix: augment the grammar with reasoning-friendly rules that let the model think before it answers. This recovered up to 10 percentage points of lost accuracy.

The practical takeaway: for reasoning-heavy tasks, use a two-pass approach. Let the model reason freely in the first call, then format the result into your schema in a second call. For classification, extraction, and data transformation, constrain away.

Provider Comparison

Here's what each provider actually supports as of March 2026:

Claude (Anthropic): Pass a json_schema in the API request alongside your prompt. Grammar-based constrained sampling with 24-hour compiled grammar caching. Available on Opus 4.6, Sonnet 4.5, and Haiku 4.5. Supports nested objects, arrays, enums, and optional fields.

GPT (OpenAI): response_format: { type: "json_schema", json_schema: {...} }. Built on Microsoft's llguidance engine. Strict mode requires all fields have explicit types. Supports $ref for recursive schemas.

Gemini (Google): response_schema parameter with response_mime_type: "application/json". Accepts OpenAPI 3.0 schema subset. Added May 2024, the earliest of the three.

All three handle flat objects reliably. Differences show up on complex schemas. The StructEval benchmark (2025) tested 18 structured formats across models. GPT-4o scored 76% average across all formats. Simple JSON/CSV/HTML hit above 90%, but complex conversions (text-to-TOML: 35.8%, text-to-Mermaid: 18.9%) still break badly. "Structured output" is solved for the common case and broken for the hard case.

The Schema-First Prompt Pattern

Whether you're using constrained decoding or prompt-only formatting, the same prompt structure works. Put the schema before the task, not after it. The model needs to know the output shape before it starts reasoning about the content.

The Prompt:

You are a data extraction assistant. You will receive raw text and extract structured information.

Output Schema (JSON):
{
  "company_name": "string - the company or organization mentioned",
  "contact_email": "string or null - email address if found",
  "services": ["string - each distinct service or product mentioned"],
  "sentiment": "positive | neutral | negative"
}

Rules:
- If a field's value is not found in the text, use null for strings and empty array for lists.
- "sentiment" must be exactly one of the three allowed values.
- Extract only what is explicitly stated. Do not infer.

Text to process:
"""
Meridian Tech Solutions (info@meridiantech.co) provides cloud migration
consulting and managed Kubernetes services. Their clients report high
satisfaction with response times and project delivery.
"""

Why This Works: The schema appears before the input text, so the model plans its output structure before processing the content. Each field includes a type hint and a brief description of what to extract. The rules section eliminates ambiguity about null handling and enumerated values, which are the two most common sources of schema violations.

Expected Output:

json { "company_name": "Meridian Tech Solutions", "contact_email": "info@meridiantech.co", "services": ["cloud migration consulting", "managed Kubernetes services"], "sentiment": "positive" }

The Retry-With-Validation Pattern

For prompt-only approaches at low volume, a validation loop beats constrained decoding in simplicity. Here's the pattern using Python and Pydantic:

The Prompt (code pattern):

from pydantic import BaseModel, ValidationError
import json

class InvoiceExtraction(BaseModel):
    vendor: str
    amount: float
    currency: str
    date: str  # ISO 8601
    line_items: list[str]

def extract_invoice(raw_text: str, llm_call, max_retries: int = 3):
    schema_description = InvoiceExtraction.model_json_schema()

    prompt = f"""Extract invoice data from the following text.
Return ONLY valid JSON matching this schema:
{json.dumps(schema_description, indent=2)}

Text:
\"\"\"
{raw_text}
\"\"\"
"""
    for attempt in range(max_retries):
        response = llm_call(prompt)
        try:
            parsed = json.loads(response)
            return InvoiceExtraction.model_validate(parsed)
        except (json.JSONDecodeError, ValidationError) as e:
            prompt += f"\n\nYour previous response failed validation: {e}\nReturn corrected JSON only."
    raise ValueError(f"Failed after {max_retries} attempts")

Why This Works: Pydantic validates both JSON structure and field types. When validation fails, the error message gets appended to the prompt, giving the model specific feedback on what went wrong. Most failures resolve on the first retry. This pattern costs an extra API call roughly 5% of the time with modern models, which is cheaper than implementing constrained decoding infrastructure for low-volume use cases.

The Instructor library (3M+ monthly downloads, supports 15+ providers) wraps this exact pattern into a single function call if you want to skip writing it yourself.

The Two-Pass Pattern for Reasoning Tasks

When you need both structured output and high-quality reasoning, split the work:

The Prompt (Pass 1 -- free-form reasoning):

Analyze this customer support ticket and determine:
1. The root cause of the issue
2. The severity (P1-P4)
3. Which team should own the resolution
4. Recommended next steps

Think through each point carefully before reaching your conclusions.

Ticket:
"""
[ticket text here]
"""

The Prompt (Pass 2 -- formatting):

Convert the following analysis into JSON matching this exact schema:
{
  "root_cause": "string",
  "severity": "P1 | P2 | P3 | P4",
  "owning_team": "string",
  "next_steps": ["string"],
  "reasoning_summary": "string - one sentence"
}

Analysis:
"""
[output from Pass 1]
"""

Why This Works: Pass 1 lets the model reason without any structural constraints. No token masking, no forced field ordering. Pass 2 is a straightforward formatting task where constrained decoding works perfectly because no reasoning is needed. You get the best of both approaches at the cost of one extra API call.

When to Use What

Prompt-only works when your schema is flat (under 5 fields), your volume is low (under a few hundred calls per day), and you can tolerate an occasional retry. This covers most internal automation and prototyping.

Constrained decoding pays off when you have nested schemas, high volume, or zero tolerance for malformed output. Use your provider's native json_schema parameter. The latency overhead is negligible with modern engines.

Two-pass generation is the right call when the task requires reasoning and the output requires strict formatting. Classification and extraction can be single-pass. Analysis, planning, and multi-step reasoning should be two-pass.

Cross-provider libraries like Instructor or PydanticAI make sense when you're calling multiple providers or want to swap models without rewriting validation logic. At single-provider scale, the native SDK is simpler.

The common mistake is reaching for the most complex solution first. Start with a well-structured prompt. Add Pydantic validation. Escalate to constrained decoding when the failure rate justifies it. Most teams never need to go past step two.

Want hands-on training on structured output patterns and production prompt engineering for your team? Connect with Kief Studio on Discord or schedule a session.

JSON prompting

Structured Output Prompting: Force Any LLM to Return Valid JSON Every Time