Structured Outputs vs Function Calling vs JSON Mode: Which One to Use and What Breaks
Use structured outputs when you want the model's reply to follow a schema, use function calling when the model must call your tools, and avoid plain JSON mode when a schema-enforcing option exists. All three return JSON, but only structured outputs and strict tool use promise that the JSON matches your schema. Both OpenAI and Anthropic still document exceptions, namely refusals and truncated answers, so your code needs a path for them. This guide summarizes both vendors' documentation as read on 10 November 2026.
Key facts
- OpenAI's guide says to use function calling when connecting the model to tools, functions and data in your system, and to use text.format when you want to structure the reply to the user.
- OpenAI states that JSON mode produces valid JSON but does not enforce your schema, and recommends always using Structured Outputs instead of JSON mode when possible.
- Anthropic offers two features: JSON outputs, set through output_config.format, and strict tool use, set with strict true. They can be used separately or in one request.
- Anthropic says outputs are produced by constrained decoding, so they are valid and type safe, except when the model refuses or the answer is cut off by max_tokens.
- Both vendors add latency the first time a new schema is used, and Anthropic says compiled grammars are cached for 24 hours from last use.
Pick by what you are building
The vendors draw the same line. If the model's job is to act, for example look up an order or book a slot, define tools and use function calling. If its job is to answer in a fixed shape, for example extract fields from an invoice or classify a ticket, use a schema on the response. Anthropic puts it as the difference between controlling what the model says and validating how it calls your functions. You can combine them: Anthropic notes a single request can call tools with guaranteed-valid parameters and also return a structured JSON response.
Why JSON mode is the weak option
JSON mode guarantees syntax, not shape. A reply can parse and still miss a required field, use the wrong type or invent a key. OpenAI recommends Structured Outputs over JSON mode wherever possible, and Anthropic's feature is described as removing JSON parse errors and schema-violation retries. If you are still validating and retrying in a loop, check whether a schema-enforcing option now replaces that code.
Schema limits to check before you commit
Enforced schemas support a subset of JSON Schema. Anthropic's page lists what is supported and what is not:
- Supported: basic types, enum values, const, anyOf, internal $ref definitions, common string formats such as date, email and uuid, and required fields. Objects must set additionalProperties to false.
- Not supported: recursive schemas, minimum and maximum, minLength and maxLength, external $ref, and array limits beyond a minimum of 0 or 1 items.
- Unsupported features return a 400 error with details. Anthropic's SDK helpers can move such constraints into the descriptions, and validation of the response is then up to your code or the helper you use.
- Anthropic also lists complexity limits per request: 20 strict tools, 24 optional parameters and 16 union-type parameters.
OpenAI's raw JSON Schema examples use required on every object and additionalProperties false. Test your real schema against each vendor you might use, because a schema that works on one may be rejected by the other.
The two cases where the schema is not guaranteed
First, refusals. OpenAI says a refusal does not necessarily follow your schema, and tells you to check the refusal field. Anthropic says a refusal returns stop_reason refusal with a 200 status, you are billed for the tokens, and the output might not match the schema. Second, truncation. OpenAI says a response with status incomplete and reason max_output_tokens may be cut off. Anthropic says stop_reason max_tokens can leave the output incomplete, and advises a retry with a higher limit. Treat both as expected states, not exceptions.
Latency, caching and token cost
OpenAI says the first request with any schema has additional latency while the schema is processed, and later requests with the same schema have none. Anthropic describes the same effect as grammar compilation, cached for 24 hours from last use, and invalidated when the schema structure changes. Anthropic adds two costs. Changing output_config.format invalidates the prompt cache for that conversation thread, and the model receives an extra system prompt describing the format, so input token counts rise slightly. If you change schemas often, expect more cache misses.
What to do
- Choose by job: tools for actions, a response schema for fixed-shape answers.
- Replace JSON mode and validate-and-retry loops with an enforced schema where the vendor supports it.
- Run your real schema through each vendor and note which constraints it rejects.
- Handle refusals and truncation explicitly, with a fallback and an alert.
- Keep schemas stable, so grammar and prompt caches stay warm.
Agent builders may also want our overview of computer-use agent architectures, and a comparison of Claude Code, Cursor and Copilot. To find developer tools, browse the AI tool directory.
Questions people ask
What is the difference between JSON mode and structured outputs?
JSON mode returns valid JSON without enforcing your schema. Structured outputs enforce the schema you supply. OpenAI recommends structured outputs over JSON mode when possible.
Does a schema guarantee valid output every time?
Not in two cases. A refusal may not follow the schema, and an answer cut off by the token limit can be incomplete. Both vendors document these exceptions.
Can I use function calling and a response schema together?
Anthropic says yes: JSON outputs and strict tool use can be used independently or in the same request.
Sources