Model Ladders: Answer With Small Models, Escalate Only When Needed

Send every request to the strongest model you can afford and you pay frontier prices for questions a small model answers correctly. Send everything to a small model and the hard requests fail quietly. A model ladder sits between the two: cheap steps handle what they can, and a large model is called only when the request needs one. The idea is simple. Doing it well depends on one hard question: how do you know, before or just after answering, that a request was too hard for the rung it landed on?

What the research shows

Two patterns dominate the literature.

Cascades try models in order of cost and stop at the first acceptable answer. FrugalGPT made the case early: API prices across providers differed by up to two orders of magnitude, and a learned cascade matched the best single model of the time with up to 98% lower cost on its benchmarks, or improved accuracy by 4% at the same cost. The scorer that decides whether to stop is the critical piece.

Routers pick one model up front. Hybrid LLM trained a router that predicts query difficulty and reported up to 40% fewer calls to the large model with no drop in response quality. RouteLLM learned routers from human preference data and reported cost reductions of more than two times in some settings without losing quality.

The two are not rivals. A unified treatment of routing and cascading derives an optimal cascading strategy, proves an existing routing strategy optimal, and combines them as cascade routing, which consistently beat either alone in its experiments. Its central finding is practical: the quality estimator, not the selection algorithm, decides how well any of this works.

What the research does not show

Recent benchmarks are a useful correction. RouterEval assembled hundreds of millions of performance records across thousands of models and concluded that most routing methods leave significant room for improvement. LLMRouterBench evaluated routers under one framework and found that many perform similarly, several recent approaches (commercial routers included) do not reliably beat a simple baseline, and a large gap to the oracle remains, mostly from failing to recall the right model. It also confirmed the premise of the whole field: models are strongly complementary.

Two lessons follow for anyone building a ladder:

1. Measure on your own traffic. A router tuned on a public benchmark knows that benchmark's difficulty distribution, not yours.
2. Close the loop. The only reliable signal that a rung was too low is what happened afterwards: the answer was used, the user retried, or the request was taken to a stronger model.

Ladders are mostly not model calls

In agent systems, the cheap rungs are often not language models at all. A routing classifier, an embedding lookup, a retrieval step, a policy or budget check, an extractor, a verifier that checks a structured answer against a schema. Each of these can run on small hardware next to the caller. The large model becomes one tool among several, called when the cheaper steps cannot settle the request.

That shifts the design question from "which model" to "which machine runs which rung". A small device close to the data can route, classify and retrieve. A GPU provider runs the large model when the orchestration escalates.

How Tenzro routes by difficulty

Tenzro's intent routing lets a caller describe what it needs instead of naming a model: a use case (chat, code, reasoning, research, summarize, extract, embed), a per-request budget, a minimum quality tier (cheap or strong), and a cost-quality knob between 0 and 1.

{
  "jsonrpc": "2.0", "id": 1,
  "method": "tenzro_routeIntent",
  "params": {
    "use_case": "code",
    "budget": "500000000000000",
    "optimize": 0.3,
    "quality_floor": "cheap",
    "prompt": "Write a Rust function that reverses a linked list."
  }
}

When the request carries a prompt and the node has a text-embedding model loaded, the prompt is placed in a difficulty cluster, an online clustering of prompts that node has routed. Each candidate is then scored on a blend of normalized cost and its observed error rate in that cluster, weighted by the knob. A cheap model that keeps being escalated on prompts like this one loses to a stronger model even though it costs less. The prompt is used only for placement; this discovery call sends it to no provider.

Clusters start empty. Until a cluster has observations, selection falls back to declared model metadata (parameter count, context window, capability tags), and unmeasured model and cluster pairs get a bounded exploration allowance, so a model is not written off before it has been tried. The decision comes back with the selected model, its tier, an estimated cost, ordered alternates, the cluster and the observed error rate, and a plain-text reason stating whether scoring was measured or declared.

The loop closes with tenzro_recordRouteOutcome. tenzro_chatByIntent, which selects and dispatches in one call, records resolved and failed from the dispatch itself. escalated is the outcome only the caller knows, because it means the caller took the answer to a stronger model, so the caller reports it. From the command line:

tenzro chat --use-case code --budget 500000000000000 --optimize 0.3 --quality-floor cheap

Two hard ceilings sit outside the scoring: a payer's spend window and, when a payer address is given, its on-chain balance. A request the payer cannot afford is refused at discovery, before any provider is dialed.

What this means for model routers and builders

A ladder pays off where many requests are easy and fails where they are not, so treat it as a measured policy rather than a default. Keep the quality estimator honest by feeding it outcomes. Expect the gap to the oracle to be about recall: the right small model existed but was not chosen. And design the cheap rungs as tools that run close to the caller, not only as smaller chat models.

Where Tenzro fits

Built: intent routing with use case, budget, quality floor and cost-quality knob; difficulty clustering from prompt embeddings with per-cluster observed error rates; outcome reporting including escalation; ordered fallbacks; and spend and balance ceilings checked before dispatch. Every served answer carries a signed receipt naming the model and billed units.

Planned: hybrid jobs in which an edge device hosts a signed harness that runs the cheap rungs locally and escalates to GPU providers, each party paid from the caller's budget against replayable receipts. The design rule for that work is that a caller never pays more through the network than through the best direct path to the same model. The network is live in preview; governance decides the steps to testnet and mainnet.