Hisham AlhussainHere's a confession: for a while, I sent everything to the biggest model available. It was easier. The answers were good, and at prototype scale, the bill didn't matter.
Then things went to production, the volume went up, and suddenly the bill mattered a lot. That's when I got serious about routing.
Most requests are easy
When I actually looked at the traffic on one of my assistants, the majority of requests were simple. Greetings, FAQs, "what are your hours," a status check. A big frontier model handles those perfectly. So does a model that costs a fraction as much and answers twice as fast.
The expensive model was earning its keep on maybe one request in five: the ambiguous ones, the multi-step ones, the ones where getting it wrong actually costs something.
How I route
There are lots of ways to do this. Here's what's worked for me:
- Route by task, not by guesswork. In an agent, different steps have different difficulty. Classifying intent or pulling fields out of a message? Small model. Planning a multi-step action or handling a tricky complaint? Big model.
- Escalate on low confidence. Let the small model try first, and escalate if it signals uncertainty or fails a validation check.
- Keep the rules simple. A routing system so clever nobody understands it is its own problem. I'd rather have three clear rules than a black box.
Caching is free money
Prompt caching is one of the most underused tricks out there. If your system prompt, tool definitions, and reference documents stay the same between requests, cache them. On long, stable prompts, the savings can be dramatic, and you get lower latency on top.
I also cache full responses for truly repeated questions where the answer doesn't change. Just be careful with anything personal or time-sensitive.
Measure the right thing
Cost per token is the wrong metric. What you care about is cost per resolved task. A cheap model that gets it wrong and needs a retry, or worse, needs a human to step in, can easily cost more than doing it right the first time with a bigger model.
So I track a few numbers together: cost per resolved conversation, resolution rate, and how often things get escalated to a human. If routing lowers cost but resolution drops, that's not a win.
Don't let savings break quality
Every routing change goes through the eval suite, same as anything else. I run the evals against the routed setup, not just the individual models, because the interactions matter. A small model that's fine on its own can still pass bad context to the next step.
The result
On the projects where I've done this properly, costs dropped a lot while quality held steady or even got a bit better, mostly because the fast model made the simple stuff feel snappier.
If you're sending everything to your biggest model, pull a week of logs and sort the requests by difficulty. I bet you'll be surprised how much of it is easy.