How to Benchmark Local LLMs with Ollama: Real Numbers, Not Hype

Every few weeks there’s another round of “this small open model matches that giant one” headlines. Qwen3.8-27B supposedly touches something close to Opus-class work. Z.ai keeps shipping GLM variants that claim better code the smaller they get. And on the provenance side, Anthropic just started weaving invisible watermarks into everything Claude writes — a fact I believe because I read it on Anthropic’s own site. The through-line is the same each time: the people selling these models can’t even agree on how to tell where text came from, yet they keep hurling benchmark claims at you.

Here’s what none of those headlines tell you: a model-card benchmark is measured on a data-center GPU someone else paid for, running a task someone else chose. Not one of those numbers tells you what the model does on your laptop, in your terminal, on your work. I lean on local models a lot — I’ve stood up a personal RAG system on Ollama and chased a 770B “local” model long enough to learn some models are anything but a laptop affair. So tonight I finally sat down and measured the models sitting in my Ollama library, on the hardware I actually own. It was one of the more useful hours I’ve spent in a while.

Below is the exact method, the raw numbers from a CPU-only WSL box, and the honest takeaway: speed and “smartest” rarely line up the way the marketing suggests.

Why a leaderboard won’t tell you what your laptop can do

A public benchmark is one number from one environment. Your laptop is another environment entirely. Memory bandwidth, quantization, thermal throttling, even whether another model is sitting warm in RAM — all of it changes the result. A 4B model that looks great on a 4090 can be brutally slow on a CPU-only machine, and a tiny 0.8B model that nobody brags about can be the most responsive thing you run.

The only number that matters is the one you produce yourself, on the hardware you actually own, on a task you actually care about. Everything else is marketing with extra steps.

What Ollama’s API already gives you for free

If you run Ollama you don’t need a third-party benchmark tool. The /api/generate endpoint returns timing fields right in the JSON response — I pulled the field list from Ollama’s API docs:

• load_duration — how long it took to load the model into memory (the cold-start penalty)
• eval_count — how many tokens it generated
• eval_duration — how long that generation took
• thinking — Qwen3-era models dump their chain of thought here, separate from the answer

Tokens per second is just eval_count divided by eval_duration. The raw fields are right there and easy to script.

The benchmark: three numbers that matter

I wanted three things out of every model, in order of how much they change your real experience:

1. Cold-start (load) time

The penalty you pay the first time you call a model that isn’t sitting warm in memory. On CPU it can dwarf the actual answer, and it decides whether a model feels instant or like a slow pause every time it unloads.

2. Throughput (tokens per second)

How fast it writes once it’s going. Token rate decides whether a long code generation feels snappy or like watching paint dry.

3. Output quality on your task

Speed is worthless if the answer is wrong, so I gave every model the same coding task and read each result line by line.

Running it: one small script

There’s no magic — and one important catch I learned the hard way. The naive way:

curl -s http://localhost:11434/api/generate \
  -d '{"model":"qwen3:4b","prompt":"write the function","stream":false}'

reads the response field and calls it done. But if your model is a Qwen3-era model, the actual reasoning can land in a separate thinking field. The first time I ran this I read an empty response with done_reason: "length" and wrongly concluded the model was broken. It wasn’t broken — it was still mid-thought, eating its entire token budget on chain-of-thought before a single visible token appeared.

So a fair benchmark has to account for thinking tokens and cap output high enough to actually let the model answer:

curl -s http://localhost:11434/api/generate \
  -d '{"model":"qwen3.5:0.8b","prompt":"write the function","stream":false,"options":{"num_predict":800}}'

Keep the prompt identical across all models, and read both response and thinking so a quiet model and an empty one don’t get confused.

The honest results from a CPU-only machine

This is the part I actually ran: Ubuntu in WSL, tonight, four models already in my Ollama library. The task asked each model to write a Python function that opens an Nginx access log, extracts the HTTP status code (the field right after the quoted request line), and returns a count per code.

Let me walk you through what these rows actually mean, because the surprises are where the value is.

Gemma3 won the quality contest. It produced a clean function that parsed the status field properly — the docstring, the extraction logic, the graceful handling of bad lines all in the right shape, at a respectable 12 tokens a second with the fastest cold load of the bunch. Middling specs on paper, best result where it counted.

Llama3.2 was fast enough but confidently wrong. Its output looked great — imports, a helper, the works — but it extracted the first numeric field on each line, which in an Nginx log is part of the remote address, not the status code. A speed benchmark alone would have shown Llama3.2 looking fine. Reading the actual output is what exposed the bug.

qwen3.5:0.8b revealed the thinking-token trap. At 32 tokens a second it was by far the fastest model here — nearly five times Gemma’s clip. But it spent 2,900 characters inside its thinking field before answering, and with my first (too-small) token cap it returned nothing at all. Give it a real budget, and it produced a sensible, well-typed answer. Fast, but it demands that you plan for its chain of thought.

qwen3:4b was the letdown nobody warns you about. The biggest model, the one you’d assume wins. On this CPU it was the slowest — about 5.4 tokens a second — and even after a 600-token budget and 131 seconds it still hadn’t surfaced a single visible answer; it was still writing chain of thought. Bigger is only better when your hardware can feed it.

What the numbers actually mean (and what they don’t)

These are honest numbers, but they’re local ones. Treat them as a reproducible method, not universal rankings you can quote elsewhere — my box, my quantizations, my task. What the exercise does prove is the process: pick a task you’d actually run, budget enough tokens for the model to think and answer, measure load time and token rate, then read the output line by line. Do that and you’ll dodge the two mistakes that waste the most time — trusting a leaderboard you can’t reproduce, and assuming the biggest model is worth the wait.

It matters twice as much when you build on top of these models. If you’re building self-improving agents on local models, a model that answers in half the time changes what your loop can do. And the cold-start penalty I keep circling is exactly what trips up naive LLM router setups — a router that treats a cold model as warm will make the wrong call, every time.

Bottom line

I started this half expecting the biggest model to crush the field and the smallest to embarrass itself. It didn’t work out either way. The biggest got stuck thinking. The smallest was blindingly fast but needed careful budgeting to say anything useful. And a middle-weight model I’d underestimated quietly wrote the best answer.

That’s the entire reason to benchmark locally instead of trusting the marketing page — the results will surprise you, and they’re the only numbers that ever predict what your machines will actually do. Ten minutes of scripting saves you from picking a model on paper and being wrong in production.

And the instinct behind all of this — verify the tool before you trust it — is the same one that kept me sane testing a local AI text detector a few days back. Treat every claim the same way: test it yourself, on your own hardware, and let the output do the talking.