Stop Including JSON in Your Prompts

A practical guide to structured outputs in LangChain v1

If your prompts still include “Please return your answer as JSON” , it’s time to stop.

tl;dr LangChain v1 refines and standardizes with_structured_output so it works consistently across providers that support it and falls back to tool calling elsewhere. It’s now built directly into the agent loop via the response_format parameter.

In most cases, you no longer tell the model to output JSON. Instead, show it a schema and let the runtime enforce it.

As an agent engineer, this is a relief: prompts stay focused on intent, and types handle structured output.

The Old Way: Prompt-Engineered JSON

A common approach looked like this:

This (mostly) worked. I hoped the model didn’t add text before the JSON ("Great insight! "). I  hoped that json.loads() didn’t need a regex bandaid. And I hoped to rememer to adjust so that the schema matched my code (it rarely did).

There’s now a cleaner, more reliable way to accomplish this.

The New Way: Schema-First, Provider-Native

with_structured_output() — for direct model calls

``

This returns a typedDecision object, not a string. Under the hood, LangChain automatically picks the right mechanism (OpenAI tool call, Anthropic function call, JSON mode, etc.).

Note: with_structured_output isn’t new—it existed before LangChain v1. What’s new is tighter integration and consistent behavior across providers and the agent loop, making it the paved road for structured outputs.

What follows is new in v1:

create_agent(..., response_format=...) — for agents

``

Here, you again receive a typedDecision object, generated directly inside the agent loop.

LangChain also supports two explicit strategies for agents: ToolStrategy (forces tool-calling) and ProviderStrategy (uses provider-native structured output). In most cases, simply passing the schema (response_format=Decision) lets LangChain automatically pick the best option.

Does this mean you can’t include JSON in your prompts? Not at all—you just rarely need to unless:

1. You’re on a model without structured-output support (some open-weights models),
2. You need partial streaming of raw text, or
3. You intentionally want looser, generative fields.

Enjoy cleaner prompts and type-safety! More langchain v1 updates to come!