Skip to content
Search
DevelopmentGlossary

Model Routing

Most traffic in a real application is easy, and a small model handles it well. Routing sends those requests to the cheap model and saves the expensive one for the requests that need it. A router can be a simple rule (by task type, input length or customer tier), a classifier, or a cascade that tries the cheap model first and escalates when the answer fails a check. FrugalGPT (2023) studied cascades of this kind, and RouteLLM (2024) trained routers on human preference data and reported cost reductions of over two times in some cases without lowering response quality.

Routing only pays if you can tell when the cheap model got it wrong, so build evals first and score both routes on the same golden dataset. An LLM-as-judge check can serve as the escalation trigger in a cascade. The supervisor-worker pattern is a close cousin, and the playbook supervisor-worker agent pattern for cost optimization shows a full setup.

Example

# A simple rule-based router with a quality check
def route(request):
    if request.task in {"classify", "extract", "short_reply"}:
        answer = call(SMALL_MODEL, request)
        if passes_checks(answer, request):
            return answer
    # Hard task, or the small model's answer failed a check
    return call(LARGE_MODEL, request)

When to use it

  • High-volume apps with a mix of easy and hard requests
  • When cost or latency matters and quality can be measured
  • Pipelines where some steps are routine and some need deep reasoning

When not to use it

  • Low volume, where the engineering costs more than the savings
  • No evals to show whether the cheap route is good enough
  • Every request is hard