Structured Outputs: Making LLMs Return Reliable JSON

Why reliable structure is hard

The moment an LLM stops talking to a human and starts feeding another system, its output stops being prose and becomes an interface. A downstream service, a database write, a workflow engine, or a UI component needs a value it can parse deterministically, and in practice that almost always means JSON conforming to an agreed schema: named fields, declared types, required keys, bounded enums. The consuming code does not read; it does json.loads and then reaches for result["status"].

The problem is that the thing producing that JSON is a probabilistic text generator. It was trained to emit likely token sequences, not to satisfy a grammar. Ask it for JSON and, left to its own devices, it will usually oblige, and then occasionally wrap the object in a Markdown fence, add a chatty preamble, emit a trailing comma, close a string it never opened, invent a field, or truncate mid-array when it hits a length limit. None of these are exotic. They are the ordinary behavior of a system whose objective function has nothing to do with parseability.

This is where the gap between a demo and a pipeline opens. In a demo you run the happy path a dozen times, it works eleven, and you ship the screenshot. A production pipeline runs the same call a million times a week, and a 98 percent success rate means twenty thousand failures, each one a dropped record, a broken UI, or a poisoned batch job. Reliability at the demo scale is an anecdote; reliability at the pipeline scale is a distribution, and the tail of that distribution is where your on-call engineer lives.

So the real question is not can the model produce JSON. It obviously can. The question is what mechanism guarantees it every time, what that mechanism actually promises, and what you put underneath it for the cases it does not cover. The rest of this article works up that ladder.

The ladder of techniques

There is a rung for every level of reliability you are willing to pay for, and the tax rises as you climb. The mistake most teams make is picking a rung by fashion rather than by the guarantee they actually need, then acting surprised when the guarantee they assumed was not the one they bought.

Read the ladder from the bottom. Each higher rung constrains the model more tightly and removes a class of failure, at some cost in latency, portability, or engineering effort. Crucially, none of the rungs removes the need for the layer that wraps all of them, which is why the last column matters as much as the first.

Two readings of this table are load-bearing. First, the rungs are cumulative, not exclusive: you do not choose tool calling or validation, you run validation around tool calling. Second, the guarantee column gets weaker the closer you read it. Every rung below constrained decoding leaves the door open to malformed structure, and every rung, including the strongest, leaves the field values probabilistic. That last point is the one section six is about.

JSON mode and tool calling

JSON mode is the first real guarantee on the ladder, and it is narrower than its name suggests. When you enable it, the provider constrains generation so the returned string parses as valid JSON: balanced braces, quoted keys, no trailing commas, no Markdown fence, no chatty preamble. That eliminates an entire family of parse failures in one switch. What it does not do is make the JSON yours. JSON mode will happily return {"answer": 42} when your code expected {"risk_score": ..., "rationale": ...}. It guarantees the language, not the sentence.

Function or tool calling closes most of that gap and is, for most teams, the default path. You define a tool with a name, a description, and a parameter schema (typically JSON Schema), and the model returns arguments shaped to fit that schema. Now the provider knows the field names, types, and which keys are required, and it targets that shape directly. The same machinery that lets a model call a weather API is the cleanest way to make it emit a structured record even when there is no API behind the tool at all; the "function" is just a schema-shaped envelope you never actually invoke. The mechanics of the tool interface itself live in the Tool Calling primer.

The practical guidance: reach for tool or function calling first, because it gives you named-and-typed structure through a well-supported interface with minimal custom code. Treat JSON mode as the fallback for providers or endpoints where tool calling is awkward. And regardless of which you pick, remember you have bought structure, not correctness, and the two layers below still have work to do.

Constrained decoding

Constrained decoding is the strongest structural guarantee on the ladder because it intervenes where the output is actually made: the token sampler. A model generates one token at a time by producing a probability distribution over the vocabulary and sampling from it. Constrained decoding compiles your schema into a grammar or a finite-state machine, and at each step it masks the tokens that would make the output impossible to complete legally, forcing the sampler to choose only from tokens the grammar still permits. If the next legal character can only be a closing brace or a digit, every other token is set to zero probability before sampling happens.

state: after  {"score":     grammar expects a number
model wants:  [ 4 ][ "high" ][ , ][ 4.7 ][ null ]
mask applied: [ ok ][  BAN  ][BAN][ ok  ][ BAN ]
sampled:      4 . 7            -> {"score":4.7
                             (illegal tokens never sampled)

Because a violating token can never be sampled, the output is guaranteed to conform to the schema syntactically. Not "usually," not "after a retry," but by construction. There is no malformed-JSON tail to catch, because malformed JSON is unreachable. This is a categorical improvement over the rungs below, which reduce the failure rate but cannot drive it to zero.

It buys structure and nothing more. The grammar knows that the status field must be one of three enum strings; it will happily let the model pick the wrong one of the three. It can force a number into a numeric field but cannot make that number correct. There is also a modest cost: computing the token mask adds a little per-step latency, and over-tight grammars can occasionally nudge the model into a worse answer by forbidding a token it "wanted" for good reasons, so the quality trade is real if small. And support is less uniform than JSON mode, which is a portability consideration if you route across providers. Constrained decoding removes an entire failure class; it does not remove the need to check what the conforming output actually says.

Validate and retry

Every rung above buys a different guarantee; validate-and-retry is the one that assumes the guarantee will eventually break and catches it when it does. The loop is deliberately unglamorous. Parse the model's output. Validate the parsed object against an explicit schema, expressed as JSON Schema or, in a Python stack, a Pydantic-style model that gives you typed fields and custom validators for free. If it passes, ship it. If it fails, feed the specific validation error back to the model and ask again, and keep a hard cap on how many times you will do that.

      call model ---> parse ---> validate --pass--> use it
         ^                          |
         |                        fail
         |                          v
         +---- re-ask with error   (attempt < max?)
                                    |
                                  no -> fallback / dead-letter

The retry message earns its keep by being specific. "That was invalid" moves nothing; "field due_date must match YYYY-MM-DD, received next Tuesday" gives the model the exact defect to fix, and correction rates on a second attempt are high precisely because the error is concrete. This is also where semantic checks live: a Pydantic validator can enforce that end_date follows start_date or that a total equals its line items, catching wrongness that no structural layer sees.

The counterintuitive rule is that this loop belongs even when you already use constrained decoding. Constrained decoding guarantees the JSON parses and matches the shape; it does not guarantee the enum is the right enum, the array is complete rather than truncated at a token limit, or the cross-field invariants hold. Validation is your semantic net, and it is the same net whether the structural rung underneath is strong or weak. Bound the retries (two or three attempts is typical) so a genuinely un-answerable input degrades to a logged fallback or a dead-letter queue instead of an unbounded, expensive loop. That bound is a cost control and a reliability control at once.

Structurally valid, semantically wrong

The trap that catches mature teams is assuming that once the JSON parses and matches the schema, the job is done. It is half done. Structural validity is necessary but not sufficient, and the failures that survive it are more dangerous precisely because they look clean. A malformed response throws an exception you notice immediately. A well-formed lie sails straight into your database and surfaces three weeks later as a support ticket nobody can explain.

The recurring shapes of valid-but-wrong output are worth naming so you can test for them:

• Hallucinated enum values where a schema without a strict enum accepts "URGENT_HIGH" when only low, medium, high are meaningful downstream.
• Coerced types where "42" arrives as a string, or a boolean is rendered as "true", and a lenient parser silently accepts it into a field the rest of the system treats as a number.
• Truncated arrays and objects where a length limit cut generation short, yet what was emitted before the cut still parses as valid JSON, so you get a complete-looking record missing half its items.
• Plausible-but-false fields where the customer name is well-formed and entirely invented, the confident kind of hallucination structure does nothing to detect.

The design consequence is to make your schema do as much semantic work as it honestly can, then accept that the remainder is a validation-and-verification problem, not a schema problem. Strict enums, tight numeric bounds, and required-field lists convert a class of semantic errors into structural ones the parser will reject for free. Everything past that boundary is application logic, and pretending otherwise is how plausible-but-false data enters systems of record.

The architect view

Step back from the individual call and the pattern is familiar: the schema is a contract. It is the agreed interface between a non-deterministic producer and the deterministic systems that consume it, and it deserves the same treatment as any other contract in your architecture: versioned, owned, reviewed on change, and tested against. This is the same discipline as spec-driven development, applied at the boundary where a probabilistic component meets everything downstream of it. Writing the schema is designing the interface, and it should happen before the prompt, not after the first parse error.

On technique, the default I hold teams to is straightforward. Prefer tool calling or constrained decoding for the structural guarantee, because both give you shape enforcement with far less custom parsing than prompt-and-pray ever will. Wrap validate-and-retry around whichever you choose, because it is the only layer that catches semantic failure and the only one that degrades gracefully when the model has a bad day. Never rely on prompt-and-pray for anything that writes to a system of record.

The through-line is the one that runs through all of context engineering. A probabilistic component in a deterministic pipeline is not a reason to lower your engineering standards; it is a reason to raise them at the seam. Structured outputs are that seam. Build the contract explicitly, enforce the structure with the right rung, validate the meaning underneath it, and make the whole apparatus a capability the organization owns rather than a snippet each team copies. Do that and the model's occasional bad JSON stops being an incident and becomes a logged, bounded, unremarkable retry.