I keep a small spreadsheet of which model does what best. Summaries go to one, code review to another, and the cheap one handles the boring bulk. It works, but maintaining that by hand is exactly the kind of chore that made me wish for a router — something that sits in front of all my models and sends each request where it belongs. Last week, Ramp announced it built exactly that, and it got me thinking about how far model routing has come, and how little of it you actually need to buy.
Model routing is becoming infrastructure
Ramp, the expense-management fintech, launched an AI model routing service it simply calls Router. It gives users an API to call different large language models through one endpoint, and Ramp says it has been running its own version internally for the past three years. The news arrived on the heels of reports that Stripe is in talks to acquire OpenRouter for more than $7 billion — the middleware layer that lets developers swap AI models without rewriting their apps.
That wave — OpenAI, Anthropic, DeepSeek, Moonshot, Minimax, Nvidia, xAI, and Z.ai are all reachable through Ramp’s Router — tells you where the industry is headed. Nobody wants to be locked to one model, and nobody wants their app to break when pricing or quality shifts. But you don’t need a $7 billion acquisition to get that flexibility. A lightweight router pointed at your own local models gives you the same idea on a scale you actually control.
What I’m building
For this walkthrough I set up a model router with LiteLLM, an open-source proxy that normalizes dozens of providers behind one API, and pointed it at a local Ollama instance already running two models on my machine: a small, fast qwen3.5:0.8b (my “cheap” tier) and a stronger gemma3 (my “smart” tier). This mirrors the same stack I used for my Ollama-based RAG pipeline, just in reverse — instead of storing knowledge, this layer decides which brain answers each question.
The commands below are the ones I actually ran, so treat them as tested rather than theoretical.
Step 1: Install LiteLLM and check your local models
LiteLLM is pip-installable. On my side it pulled in version 1.97.0 without drama.
pip install litellm
ollama list
The ollama list output is your “model inventory” — the thing a router needs to know about before it can make decisions. For this demo I used qwen3.5:0.8b and gemma3, but any two models you have locally will do the job.
Step 2: Define the model list
The core of LiteLLM’s in-process Router is a model_list: a list describing each model and the provider details needed to reach it. Each entry gets a friendly alias — I used local-fast and local-strong — plus the provider name and the local API base.
from litellm import Router
router = Router(model_list=[
{"model_name": "local-fast",
"litellm_params": {"model": "ollama/qwen3.5:0.8b", "api_base": "http://localhost:11434"}},
{"model_name": "local-strong",
"litellm_params": {"model": "ollama/gemma3", "api_base": "http://localhost:11434"}},
])
Notice how nothing here is hardcoded to one provider. The same Router accepts OpenAI, Anthropic, or any of the dozens LiteLLM supports — you just add a block to model_list. That single abstraction is the whole point of a router: your application talks to local-fast or local-strong, and the routing layer handles the plumbing.
Step 3: Route a request
With the router built, a request is one call. I sent a quick “ROUTE OK” prompt to the strong model and it came back correctly through ollama/gemma3 in about 12 seconds on my machine.
resp = router.completion(
model="local-strong",
messages=[{"role": "user", "content": "Reply with exactly: ROUTE OK"}],
)
print(resp["choices"][0]["message"]["content"])
Nothing about that call mentions Gemini or Ollama by name. The router resolved local-strong to the right provider behind the scenes. Swap the model list tomorrow and the app code doesn’t change — that’s the whole flex.
Step 4: Add a routing strategy
Where Ramp’s Router gets interesting is its “strategies”: route only hard problems to expensive models, pick a model by user-set benchmarks, or prefer certain flex pricing tiers. You can build the same idea in a few lines — a function that looks at the prompt and picks an alias, then hands it to the router.
def pick_model(prompt):
hard = len(prompt) > 60 or any(
k in prompt.lower() for k in ["explain", "why", "code", "math", "compare"]
)
return "local-strong" if hard else "local-fast"
In my test, a one-line “Say hi” auto-routed to the cheap local-fast model, while a longer reasoning question (“Explain why model routing matters for cost and latency and compare two strategies”) landed on local-strong. A real strategy would use smarter signals — token budget, required latency, a confidence threshold — but the skeleton is exactly this.
What I learned about local routing latency
The honest part: on my test box the “cheap fast” model took 163 seconds while the “smart” one answered in 13.8 seconds. That sounds backwards until you realize the small model was a cold start — it had to load into memory before answering, and model warm-up easily dwarfs the difference between a 0.8B and a 4B model. If you’re routing locally, the biggest win isn’t choosing the smallest model; it’s keeping models loaded and matching them to load. Exactly the kind of thing that pushed me toward the self-host ethos I wrote about when I set up Immich to replace Google Photos.
The same router, pointed at the cloud
Here’s the part that makes the pattern stick. Because model_list is just a list of provider configs, adding a hosted model is the same shape as adding a local one — you append a block and give the router an API key for that provider. One router can mix your local gemma3 with a frontier model from OpenAI or Anthropic, then let your strategy decide when a request is worth paying for.
{"model_name": "cloud-premium",
"litellm_params": {"model": "openai/gpt-5",
"api_key": os.environ["OPENAI_API_KEY"]}},
I didn’t wire up a paid key for this test, so I’m not going to pretend I benchmarked it — but the config shape is identical, and that’s the point. Your application never needs to know whether a request landed on a $0.0002 local token or a frontier one. The router redraws that line without touching your code, which is precisely why inference middleware is suddenly worth billions: it turns every AI cost decision into a config change instead of a rewrite.
Pitfalls I hit
A couple of gotchas worth flagging so you don’t chase them. First, LiteLLM’s full web proxy server is a separate install — pip install 'litellm[proxy]' — and on my setup it tripped over a FastAPI version mismatch on startup. The in-process Router I used in this guide sidesteps all of that, so start there. Second, install the CPU build of heavy modeling dependencies if you’re on a box without a GPU — my AirLLM tutorial walks through that pattern.
When a hosted router makes sense
None of this means you should roll your own routing in production. OpenRouter’s likely $7 billion price tag is a clue that there’s real complexity in reliability, billing, load balancing, and uptime — the stuff I wrote about when I covered Stripe’s move for OpenRouter. And Ramp’s own Router comes with a caveat worth reading: its default data retention logs inputs and outputs for a year (it says it strips personal identifiers before using them to improve the product), and you have to opt out. If you’re routing sensitive prompts, that trade-off decides the answer for you — local routing keeps every prompt on your own hardware.
For my use case, a LiteLLM router over local models is the middle path that fits. I get one API for every model, a strategy knob for cost, and none of my prompts leave the machine. If you do lean on rented GPUs for heavier work, the same “check before you commit money” discipline I covered in my ComputeFence pre-flight guide applies here too — a router just decides which model answers, it doesn’t tell you whether the underlying instance is worth paying for. It’s the same instinct that made me build a simple RAG pipeline with Ollama and Gemma 3 in the first place — own the stack, stay flexible, and don’t rent more than you need.
Whether model routing becomes a multi-billion-dollar middleware category or a quiet part of every local setup, the capability is now small enough to build in an afternoon. And unlike the hand-maintained spreadsheet, a router doesn’t forget which model hates which task.