How to Get Reliable JSON From Local LLMs With Ollama

Last week, I rebuilt a small internal tool at the office and added a local AI helper to it. The helper’s job was simple: read a vendor report and hand my script a clean list of items with prices and dates. It should have taken an hour. What actually happened is that the model kept wrapping my JSON in pleasant little sentences, dropping braces, and once even decided to answer “sure!” instead of returning data. That’s the whole reason I reached for a different approach — and it’s what this guide is about.

If you’ve played with local models through Ollama, you know they’re brilliant at prose but sloppy when you need machine-readable output. The fix isn’t to nag the model with a longer prompt. It’s to use the two features Ollama gives you for exactly this job: structured JSON output and tool (function) calling. Both are baked into the Ollama API, and both work with small models running on a normal laptop. I tested every command below on my own box with qwen3:4b, so these aren’t theoretical.

Why local models mangle JSON

A language model isn’t a JSON parser. Left to its own devices, it predicts the most likely next word — and for a chat model, the most likely next word after “here are the results” is more chat, not a closing brace. Ask a model to extract restaurant details and you might get a paragraph, a list, or valid JSON. It depends on the model, the phrasing, even the temperature.

That’s the core problem. Downstream automation needs predictable output. My script either gets a clean JSON object it can parse, or it crashes. So instead of hoping, I use the API’s format flag to lock the output into a strict grammar, and the tools field when I want the model to choose a function at all.

Structured JSON output with the format flag

Ollama’s chat endpoint accepts a format: "json" parameter. When set, the model is constrained to produce valid JSON — no extra commentary outside the object, no partial braces. Here’s the request I ran to extract a restaurant’s details:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:4b",
  "messages": [{"role": "user", "content": "Write ONLY valid JSON with the keys name, rating and cuisine. Cafe Lumen in Roxas City, 4.5 out of 5, Filipino and Italian fusion."}],
  "stream": false,
  "format": "json"
}'

The response came back as clean, parseable JSON (the repo and official Ollama repository document the format flag in more depth):

{"name": "Cafe Lumen", "rating": 4.5, "cuisine": "Filipino and Italian fusion"}

Notice what happened — and what didn’t. There’s no “Sure, here’s the output”, no trailing explanation, no missing comma. I ran this exact call against my local Ollama and piped the result straight into Python’s json.loads() and it parsed on the first try. That’s the whole point.

What format: json guarantees (and doesn’t)

Let me be precise, because this trips people up. The format: "json" flag forces the structure to be valid JSON. It does not enforce a particular schema. The model decides the keys. If your script expects name, rating, and cuisine but the model decides to return {"restaurant": "...", "stars": 4.5}, that’s still valid JSON — and still won’t parse correctly in your code.

So the reliable pattern is two layers:

• Layer one: format: "json" so the output is always valid JSON.
• Layer two: a clear, explicit instruction in the prompt naming the exact keys, plus a validation step in your code that checks the keys actually exist before you trust the data.

I put the key names right in the system prompt for every extraction task. It costs nothing and removes the most common failure.

Tool calling: let the model pick a function

The second feature I relied on is function calling. Here the role flips — instead of “give me JSON,” you tell the model what tools exist and let it decide to call one. This is the same kind of decision routing I explored when I built my own model router with LiteLLM. This is how local agents do things like check the weather, query a database, or fetch a value, without hardcoding the logic into every prompt.

The request declares a tool with a name, description, and JSON-schema parameters:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:4b",
  "messages": [{"role": "user", "content": "What is the weather in Roxas City? Use the get_weather tool."}],
  "stream": false,
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get current weather for a city",
      "parameters": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"]
      }
    }
  }]
}'

Instead of generating a final text answer, the model returns a tool_calls array — a structured instruction your code can execute and feed back into the conversation:

"tool_calls": [{
  "id": "call_n9d2mnl8",
  "function": {"name": "get_weather", "arguments": {"city": "Roxas City"}}
}]

That’s exactly the output my script needs: a parsed function name and a typed arguments object. I run the real get_weather, get the temperature, send it back as a tool role message, and let the model turn it into a friendly answer. Tested live here with qwen3:4b — it correctly identified the tool and filled in the city argument without any prompting beyond the one line.

A minimal tool-calling loop in Python

Putting it together, the flow is a small loop rather than a single call:

1. Send the user’s message plus the tools list.
2. If the response has tool_calls, execute each one and append the results back as a tool message.
3. Repeat until the model returns a plain content answer.

A stripped-down version:

import json, urllib.request

def ask(messages, tools=None):
    body = {"model": "qwen3:4b", "messages": messages, "stream": False}
    if tools: body["tools"] = tools
    req = urllib.request.Request(
        "http://localhost:11434/api/chat",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        return json.loads(r.read())

messages = [{"role": "user",
  "content": "What is the weather in Roxas City? Use the get_weather tool."}]
reply = ask(messages, tools=TOOLS)

for call in reply.get("message", {}).get("tool_calls", []):
    fn = call["function"]["name"]
    args = call["function"]["arguments"]
    # Execute the real function, then feed the result back:
    messages.append({"role": "tool", "name": fn, "content": '{"temp": 29, "desc": "partly cloudy"}'})

# Now get the model's final answer using the tool result:
final = ask(messages)
print(final["message"]["content"])

For a complete, robust version you’ll add a loop guard (a max iteration count so a misbehaving model can’t loop forever), error handling around urlopen, and a check that the tool result is itself valid JSON. Those three additions turn a demo into something you’d actually trust in production.

What actually matters in practice

Running this locally taught me a few things worth passing on, in case you adopt the same stack.

• Use format: json for extraction, tools for actions. If you just need structured data out of free text, the format flag is the simpler, more reliable lever. Tool calling shines when the model must choose between actions.
• Validate everything your script consumes. A parsing error after the fact is still an error. Check required keys and types before using the value.
• First reply can be slow on a CPU-only box. My first call pulled the model into memory and took a while — later calls ran quick. Don’t judge a model’s speed by its first cold call; I covered honest local benchmarks separately in my local LLM benchmarking guide with real numbers.
• Keep the schema small. Tiny models handle a one-or-two-function schema much more reliably than a sprawling one. Add complexity gradually and re-test each time.

Wrapping up

Getting a local LLM to return reliable structured output isn’t about brute-forcing a cleverer prompt. It’s about using the two switches the API actually gives you — format: "json" to lock the grammar, and tools to route the model into calling real functions. Both let a small model behave like a dependable component instead of a chatty sidekick.

This is the same thinking I leaned on when I built a local AI text detector — output you can trust — and it pairs naturally with rounding out your own local LLM setup with a small RAG system. Once your scripts can trust what the model returns, a surprising amount of boring automation becomes possible with a laptop and a couple of GBs of disk.