Routing Queries to Models: The Cost Decision Tree Nobody Writes Down

Your biggest cost lever is not inference optimization. It's routing. Not every query needs your frontier model. Most don't. Route the easy ones to a cheap model and save 70% per query.

Most products ignore routing because it's not glamorous. You can't blog about it. You can't benchmark it. You can't claim "we use GPT-4o" if 80% of your queries run on a 13B model. So teams skip it, pay 3x what they should, and wonder why they're unprofitable.

Routing is one layer of the six-layer system that separates products that scale profitably from ones that die. This is the layer that decides whether your unit economics work.

The Query Distribution Problem

Your query volume is not homogeneous. Some queries are trivial; some are hard. A support chatbot answering "what's your return policy?" is a different problem from "how do I use feature X in context of my use case Y?"

The first needs retrieval + a 13B model. The second needs reasoning + a 70B model.

If you route everything to the 70B model:

• Easy query (retrieval + 13B solves it): $0.08 cost
• You send it to 70B instead: $0.60 cost
• You waste $0.52 per easy query

With 1,000 queries per day, 80% easy:

• Bad routing: 800 × $0.60 + 200 × $0.60 = $600/day
• Good routing: 800 × $0.08 + 200 × $0.60 = $184/day
• Savings: $416/day = $150K/year

That's not optimization. That's just not paying for things you don't need.

Measuring Your Query Distribution

Before you design routing, you need data. What percentage of your queries are easy, medium, hard? This is a model evaluation problem: you have to measure which queries the cheap model actually handles before you can classify them.

Week 1: Baseline Run everything on your frontier model (or your current setup). Log every query. At the end of the week, manually classify 100 random queries: easy (took < 500ms, first token was correct), medium (took 1-2s, required reasoning), hard (took > 2s, required planning across multiple steps).

You'll find something like:

• Easy: 65%
• Medium: 25%
• Hard: 10%

These numbers are your routing target. You want to run 65% on the cheap model, 25% on the medium model, 10% on the frontier.

The Routing Decision Tree

Once you know your distribution, build the routing logic. Start simple; refine based on feedback.

Version 1: Heuristic Rules (week 1)

if query_length < 50 tokens:
  route to cheap_model (13B)
elif "help me" or "how do I" in query:
  route to cheap_model
elif query contains technical_keyword_list:
  route to medium_model (35B)
else:
  route to frontier_model

This is crude but catches the pattern. Measure accuracy: "did the routed model succeed?"

Version 2: Learned Classifier (week 4)

Collect 500 labeled examples from Version 1 feedback. Train a small classifier (logistic regression, decision tree, or small neural net). Use that instead of hand-coded rules.

Input: query embedding + query length + keyword presence + context length Output: probability model_X succeeds; route to cheapest model above threshold

Accuracy improves. Cost stays low.

Version 3: Cost-Aware Routing (week 8)

Now factor in cost, not just success probability. A medium model that succeeds 95% of the time costs $0.20. A frontier model succeeds 99% and costs $0.60.

For this query:

• Medium model: 0.95 × (query_solved) - 0.05 × (failure_cost) = value
• Frontier model: 0.99 × (query_solved) - 0.01 × (failure_cost) = value

Route to whichever has higher expected value. Sometimes you'll route to the cheaper model even if success rate is lower, because the cost difference matters more.

Real Costs (Current Pricing, Mid-2026)

These numbers are the raw material of AI compute economics: the gap between models is where routing pays for itself.

For a typical query (1,000 output tokens):

• 13B: $0.10
• 35B: $0.20
• Frontier: $0.60-$1.20

If 80% of queries succeed on 13B, you save:

• 800 queries × $0.50 (difference between 13B and frontier) = $400/day per 1,000 queries

Scale to 100K queries/day and routing alone cuts your costs by $40K/day. That's not optimization; that's core business model.

The Feedback Loop (How to Improve)

Week 1: Log Everything

• Query text
• Query embedding (for similarity analysis)
• Routed model
• Success (did the model answer correctly?)
• User feedback (did they accept the answer?)
• Actual cost

Week 2: Analyze Failures Which queries routed to the cheap model and failed?

Pattern: queries with "code" in them fail 40% of the time on 13B. Action: add "code" to the keywords that route to medium model.

Pattern: queries > 200 tokens fail 35% of the time on 13B. Action: route queries > 200 tokens to medium or frontier.

Week 3: Update Routing Logic Adjust the classifier or rules based on failure patterns.

Week 4: Measure Improvement Success rate on cheap model: was 85%, now 91%. Cost per successful outcome: was $0.12, now $0.09. Cost savings: $0.03 × 100K queries/day = $3K/day.

Repeat monthly. Each iteration improves accuracy without increasing cost.

When the Cheap Model Fails

Routing is not just about picking the cheapest model that might work. It's about knowing when it didn't. A cheap model that fails silently is worse than no routing at all, because you ship a wrong answer at a discount. This is why verification sits next to routing: you need a check that catches the failure, then a fallback that escalates to the next model up.

The fallback rule is simple. If the routed model fails its check and cheaper models remain exhausted, escalate. If the query needs multi-step planning or tool use rather than a single answer, route it to agents instead of trying to force it through a one-shot model. Retrieval-heavy queries that keep failing usually don't need a bigger model at all — they need a retrieval architecture that works feeding the cheap one better context.

The Mature State: Multi-Model Orchestration

Once you've been routing for 3 months, you'll have:

1. A classifier that predicts success rate per model
2. Cost data per query category
3. User feedback on answer quality per route

The mature routing looks like:

for each query:
  for each available_model in [cheap, medium, frontier]:
    success_probability = classifier.predict(query, model)
    expected_value = success_probability * query_value - model_cost
  route to model with highest expected_value
  if any model fails and cheaper models remain:
    fallback to next cheapest model

This is the state where you're squeezing maximum value from your model portfolio. Cost per successful outcome is optimized. Quality stays high. Users don't know they're routed to different models; they just get answers.

The Checklist (Before You Ship)

• You've measured your query distribution (easy/medium/hard percentages)
• You've designed the initial routing heuristics (don't overthink it; v1 is 80% of the value)
• You're logging every query, routed model, and success/failure
• You have a process to analyze failures weekly and adjust routing
• You've calculated the cost per model and the cost difference per category
• You have a fallback: if the routed model fails, what's the next cheapest option?
• You've tested routing on 100 manual queries and know the accuracy
• You've factored routing cost into your unit economics model

The Bottom Line

Routing is the easiest 40% cost reduction you'll ever ship. It's not sexy. It's not a benchmark win. It's just not paying for things you don't need.

Design it before you're at 10x usage. Measure it from day one. Iterate on it monthly. By month six, routing alone will have cut your costs in half and unlocked your unit economics.

Then optimize inference. Then distill models. You've already won on the biggest lever. Everything else is tuning.