TL;DR: If json.loads() on model output fails only occasionally, there are three distinct causes: truncation, stray prose or code fences around the JSON, and schema drift (valid JSON, wrong shape). Provider-side structured outputs (constrained decoding) eliminate the second and third for supported schemas; a validate-and-retry loop with a typed model covers everything else, including the first. In production you usually want both.
What does the failure actually look like?
The symptom that sends people to a search engine is usually one of these:
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 140 (char 139) pydantic_core._pydantic_core.ValidationError: 1 validation error for Ticket priority Input should be less than or equal to 4 [type=less_than_equal, ...] They look alike but have different fixes:
| Symptom | Real cause | Right fix |
|---|---|---|
Expecting value: line 1 column 1 | Model wrapped the JSON in a code fence or added "Here is the JSON:" | Constrained decoding, or extract the outermost braces |
Unterminated string / Expecting ',' delimiter near the end | Output hit max_tokens mid-object (truncation) | Raise the limit, shrink the schema, check the stop reason |
| Parses fine, then your code crashes on a missing key or odd enum value | Schema drift: valid JSON, wrong shape | Typed validation, or a strict schema on the provider side |
The first thing to do before changing any prompt: log the raw response and the stop/finish reason for every failure. Most "the model is flaky" reports I have seen turned out to be truncation, which no prompt wording can fix.
Takeaway: classify the failure from the raw text and the stop reason before you touch the prompt.
Step 1: Validate with a typed model instead of json.loads
Parsing and validating in one step gives you an error message precise enough to feed back to the model. This uses Pydantic v2 and works with any provider, because call_model is just a function you supply.
from typing import Callable, Literal from pydantic import BaseModel, Field class Ticket(BaseModel): category: Literal["bug", "billing", "feature_request"] priority: int = Field(ge=1, le=4) summary: str = Field(min_length=1, max_length=200) def extract_json_object(raw: str) -> str: """Strip code fences / chatter by taking the outermost braces.""" start, end = raw.find("{"), raw.rfind("}") if start == -1 or end <= start: raise ValueError("no JSON object found in response") return raw[start : end + 1] def parse_ticket(raw: str) -> Ticket: return Ticket.model_validate_json(extract_json_object(raw)) The brace extraction is a blunt tool: it will not rescue truncated output, and it can mis-slice if the model emits two objects. That is fine, because the retry loop below catches those cases.
Step 2: Retry with the error, not with the same prompt
Resending the identical request is a coin flip. Sending the validation error back gives the model something concrete to correct. In Pydantic v2, ValidationError subclasses ValueError, so one except covers both parse and schema failures.
Message = dict[str, str] def extract_ticket( call_model: Callable[[list[Message]], str], prompt: str, max_attempts: int = 3, ) -> Ticket: messages: list[Message] = [{"role": "user", "content": prompt}] last_error: ValueError | None = None for _ in range(max_attempts): raw = call_model(messages) try: return parse_ticket(raw) except ValueError as exc: last_error = exc messages += [ {"role": "assistant", "content": raw}, { "role": "user", "content": ( "That response failed validation:\n" f"{exc}\n" "Reply with only a corrected JSON object." ), }, ] raise RuntimeError(f"no valid output after {max_attempts} attempts") from last_error Two details that matter. Cap attempts at 2–3: each retry costs a full round trip and re-sends the growing conversation, so an uncapped loop turns one bad input into a bill spike. And if the failure was truncation, retrying with the same max_tokens will truncate again, so check the stop reason and raise the limit instead of looping.
Takeaway: a retry that includes the exact validation error converges; a retry that repeats the prompt just re-rolls the dice.
When do provider-side structured outputs replace the loop?
Most major providers now offer some form of schema-constrained output: OpenAI's Structured Outputs with a strict JSON schema, tool/function calling with an input schema on Anthropic's API, and grammar- or schema-constrained decoding in local runtimes such as llama.cpp and in libraries like Outlines. The model's token choices are restricted so the output cannot violate the schema's syntax. As of late 2026 the details differ per provider, so check current docs for supported schema features.
What this buys you: no code fences, no prose preamble, no invalid enum values, no missing required keys. If you want this without writing the plumbing yourself, OpenAI's Structured Outputs is the option that guarantees schema adherence on a strict schema, at the cost of being tied to that API.
What it does not buy you:
- Truncation is still possible. If generation stops at the token limit, the object is still cut off. Strict mode does not make that safe.
- Only a schema subset is supported. Constraints like
minLength,maximum, or deeply recursive structures are often unsupported or ignored, so yourField(ge=1, le=4)check may still need to run in your own code. - Syntax is not semantics. A schema-valid
priority: 4can still be the wrong priority. Constrained decoding makes the shape certain, not the content correct. - First-request latency. Some providers compile the schema on first use and cache it, so a rarely used schema can be slower on its first call.
Takeaway: constrained decoding guarantees shape, not truncation-safety and not correctness, so keep your own validation behind it.
Which approach should you use?
| Situation | Use |
|---|---|
| Hosted model, simple flat schema | Provider structured outputs plus Pydantic validation |
| Provider lacks strict mode, or schema uses unsupported constraints | Validate-and-retry loop |
| Local model (llama.cpp, Ollama-style runtimes) | Grammar/schema-constrained decoding if your runtime supports it, retry loop as backstop |
| Long outputs (big arrays, many records) | Split into smaller calls; truncation risk grows with output length |
FAQ
Why does the LLM return invalid JSON only sometimes?
Sampling is probabilistic, so a model that follows your format 98% of the time will still occasionally add a code fence, stop mid-object at the token limit, or pick an off-schema value. Intermittent failures are normal without constrained decoding.
Does asking for JSON in the prompt guarantee valid JSON?
No. A prompt instruction raises the odds but guarantees nothing; only schema-constrained decoding restricts the output tokens, and even then truncation can break the object.
How many retries should I allow for LLM JSON parsing?
Two or three. If the third attempt still fails, the cause is usually truncation or an ambiguous schema, and more retries will not help.
Bottom line
If you call a hosted model with a simple schema, turn on the provider's strict structured output and keep a Pydantic model behind it for the constraints the schema cannot express. If you are on a provider or local model without it, the validate-and-retry loop above is enough for most small-team workloads. Either way, log the raw response and stop reason so you can tell truncation from drift. Cap retries, because the loop's failure mode is cost, not correctness.
