Getting Reliable JSON Out of an LLM

The demo works. You ask the model for JSON, it returns JSON, you JSON.parse it and render a nice card. Then you ship, and somewhere around request 200 you get:

Sure! Here's the JSON you asked for:

```json
{"title": "Widget", "price": 12.99}
```

Let me know if you'd like any changes!

JSON.parse throws. Your route returns a 500. A user sees a spinner that never stops.

The fix isn't a better prompt. It's treating model output the way you'd treat any other untrusted input: parse it, validate it against a schema, and have a defined behaviour for when it doesn't conform.

The four layers

Reliable structured output is four things stacked, cheapest first:

1. Constrain generation so malformed output is unlikely
2. Extract the JSON from whatever wrapper it arrived in
3. Validate against a schema you control
4. Repair once, then fail loudly

Skip any layer and you're relying on the model behaving. It usually will. "Usually" is the problem.

Layer 1: constrain generation

Every major provider now has a way to force valid JSON. Use it — it's free and it eliminates the entire class of syntax errors.

With the Claude API, the reliable route is a tool definition. You're not really calling a tool; you're using the tool schema as an output contract, and the API guarantees the arguments match it:

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

const extractProduct = {
  name: "extract_product",
  description: "Record the product details found in the page text.",
  input_schema: {
    type: "object",
    properties: {
      title: { type: "string" },
      price: { type: "number", description: "Numeric price, no currency symbol" },
      currency: { type: "string", enum: ["USD", "EUR", "GBP", "INR"] },
      in_stock: { type: "boolean" },
    },
    required: ["title", "price", "currency", "in_stock"],
  },
} as const;

const res = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 1024,
  tools: [extractProduct],
  tool_choice: { type: "tool", name: "extract_product" },
  messages: [{ role: "user", content: pageText }],
});

const block = res.content.find((b) => b.type === "tool_use");
const data = block?.input;

tool_choice pinned to a specific tool is the important line. Without it the model can decide to just reply with prose. With it, the only legal output is a call to that tool.

If you're on a local model through Ollama, you get the same idea via format:

const res = await fetch("http://localhost:11434/api/chat", {
  method: "POST",
  body: JSON.stringify({
    model: "qwen3:14b",
    format: schema,       // a JSON Schema object, or the string "json"
    stream: false,
    messages: [{ role: "user", content: pageText }],
  }),
});

Ollama constrains the sampler with a grammar built from the schema, so tokens that would break the structure simply can't be emitted. Smaller models benefit from this far more than large ones — a 7B model asked politely for JSON is a coin flip; the same model with a grammar is deterministic in shape.

Layer 2: extract

Even with constrained generation you'll sometimes be parsing free-form text — an older endpoint, a model behind a proxy that drops the format field, a provider fallback like the one in the Ollama → OpenRouter guide. So keep a tolerant extractor:

/** Pull the first JSON object or array out of a possibly chatty response. */
export function extractJson(text: string): unknown {
  const fenced = text.match(/```(?:json)?\s*([\s\S]*?)```/);
  const candidate = (fenced ? fenced[1] : text).trim();

  try {
    return JSON.parse(candidate);
  } catch {
    // Fall back to the outermost brace/bracket pair.
    const start = candidate.search(/[[{]/);
    const end = Math.max(candidate.lastIndexOf("}"), candidate.lastIndexOf("]"));
    if (start === -1 || end <= start) throw new Error("No JSON found in response");
    return JSON.parse(candidate.slice(start, end + 1));
  }
}

Two rules keep this from becoming a monster:

• Do not write a JSON repair parser. Trailing commas, single quotes, unescaped newlines inside strings — every fix you add invites the next one, and the whole pile is dead code the day you turn on constrained generation. If the braces don't balance, that's a job for layer 4.
• Do not eval it. Obvious, but it comes up every time someone hits a single-quoted key.

Layer 3: validate

Parsing tells you the syntax is legal. It tells you nothing about whether price is a number, a numeric string, null, or missing entirely. Models are especially fond of returning "price": "12.99" and "in_stock": "yes".

If the project already has Zod, use it. If it doesn't, don't add a dependency for one schema — a hand-written guard is a dozen lines and has no version to keep up with:

export interface Product {
  title: string;
  price: number;
  currency: "USD" | "EUR" | "GBP" | "INR";
  inStock: boolean;
}

const CURRENCIES = ["USD", "EUR", "GBP", "INR"] as const;

export function parseProduct(raw: unknown): { ok: true; value: Product } | { ok: false; error: string } {
  if (typeof raw !== "object" || raw === null) return { ok: false, error: "not an object" };
  const r = raw as Record<string, unknown>;

  const title = typeof r.title === "string" ? r.title.trim() : "";
  if (!title) return { ok: false, error: "title missing or empty" };

  // Coerce the common near-misses rather than rejecting them.
  const price = typeof r.price === "number" ? r.price : Number(String(r.price ?? "").replace(/[^0-9.]/g, ""));
  if (!Number.isFinite(price) || price < 0) return { ok: false, error: "price is not a valid number" };

  const currency = String(r.currency ?? "").toUpperCase();
  if (!CURRENCIES.includes(currency as Product["currency"])) {
    return { ok: false, error: `currency must be one of ${CURRENCIES.join(", ")}` };
  }

  const inStock = typeof r.in_stock === "boolean" ? r.in_stock : /^(true|yes|1)$/i.test(String(r.in_stock));

  return { ok: true, value: { title, price, currency: currency as Product["currency"], inStock } };
}

Note what this does and doesn't coerce. "12.99" becomes 12.99 because that's an unambiguous formatting difference. A missing title is not filled with a default, because inventing data is worse than failing. The line between the two is: coerce representation, never invent content.

Returning a result object instead of throwing matters for the next layer — you need the error message to feed back to the model.

Layer 4: repair once

When validation fails, you have exactly one useful move: show the model its own output and the specific complaint, and ask again. This works surprisingly well, because the model isn't confused about the task, only about the format.

export async function extractWithRepair(pageText: string): Promise<Product> {
  const first = await callModel(pageText);
  const parsed = parseProduct(extractJson(first));
  if (parsed.ok) return parsed.value;

  const second = await callModel(pageText, {
    priorOutput: first,
    complaint: parsed.error,
  });

  const retry = parseProduct(extractJson(second));
  if (retry.ok) return retry.value;

  throw new Error(`Extraction failed after repair: ${retry.error}`);
}

One attempt, not a loop. A retry loop on a model that has misunderstood the schema burns tokens at full price to produce the same failure three more times, and turns a 2-second request into an 8-second one before the user gets their error. If the first repair doesn't land, the problem is your schema or your prompt, and no amount of retrying fixes either.

The repair message should be blunt and include both halves — what it said and what was wrong with it:

const repairPrompt = `Your previous response was rejected.

Your output:
${priorOutput}

Problem: ${complaint}

Return only the corrected JSON object. No explanation, no markdown fence.`;

Streaming and structured output don't mix well

If you're streaming (see streaming LLM output over SSE), remember you can't validate a partial object. Two options:

• Don't stream structured output. For extraction tasks the payload is small and the latency difference is a few hundred milliseconds. Just wait for the whole thing.
• Stream the prose, batch the data. If a response has both — an explanation and a structured result — stream the explanation for perceived speed and send the validated object as a final event.

What you should not do is parse partial JSON on the client to render fields as they arrive. It's a lot of fragile code to save 300ms, and it breaks the moment a repair attempt kicks in.

Test the failure path, not the happy path

The happy path is the one you'll accidentally test a hundred times during development. The paths that will actually break in production are the ones you have to construct deliberately. Feed your validator, directly, without calling a model at all:

// The five failures that actually happen in production.
const cases = [
  'Sure! ```json\n{"title":"X","price":9,"currency":"USD","in_stock":true}\n```',  // chatty wrapper
  '{"title":"X","price":"$9.00","currency":"usd","in_stock":"yes"}',              // stringly typed
  '{"title":"","price":9,"currency":"USD","in_stock":true}',                      // empty required field
  '{"price":9,"currency":"USD","in_stock":true}',                                 // missing field
  '{"title":"X","price":9,',                                                      // truncated (hit max_tokens)
];

The last one is the one people forget. When a response hits max_tokens mid-object you get syntactically invalid JSON, and no amount of prompting prevents it — check stop_reason and raise your token limit.

What to keep

• Constrain generation with tools or a grammar. It's the highest-leverage line of code here.
• Extract tolerantly, validate strictly, coerce representation but never invent content.
• Repair exactly once with the specific error, then fail with a real message.
• Log every rejection with the raw output. The pattern in those logs is what tells you the schema is wrong — a field the model keeps getting "wrong" is usually a field you described ambiguously.

The goal isn't a model that never misbehaves. It's a pipeline where misbehaviour is a handled case instead of a 500.

Related: Working with APIs · Cancelling LLM streams · The Ollama → OpenRouter fallback