Route, Don't Pick: Running Multiple LLMs Across a Development Workflow

Route, Don't Pick: Running Multiple

LLMs Across a Development Workflow The question we get asked most often is some version of "which LLM is best right now?" It is the wrong question, and answering it honestly makes that clear. The best model for planning a system architecture is not the best model for writing a hundred commit messages a day, and neither is the one you want classifying support tickets at scale. A working development team touches a dozen different kinds of work before lunch. Picking one model to cover all of them means you either overpay for the easy tasks or underdeliver on the hard ones.

The better mental model is orchestration. You stop treating model choice as a single decision and start treating it as a routing problem: every task category flows to the model that handles that category well, with fallbacks when a provider stumbles and a way to measure whether the routing was right. This post lays out how we think about that split, which models fit which category as of early August 2026, and the plumbing that makes it work in production.

Why one model is the wrong default

Two forces push against the single-model habit.

The first is cost. The spread between tiers is not small. A cheap production model sits around one to five dollars per million tokens. A frontier model can run ten to fifteen times that on output. If your agent loop reads a file, writes a patch, and runs a test on every iteration, and you point all of that at a frontier model, the bill compounds quietly until it does not. There is a widely circulated story from this year about a team that ran up a runaway frontier bill on work that an open-weight model could have handled at a fraction of the cost. The lesson is not "use cheap models." It is "match the model to the difficulty of the call."

The second force is capability shape. Models are not uniformly better or worse than each other. One tops the agentic coding benchmarks and is mediocre at long-context retrieval. Another leads scientific reasoning and is overkill for extracting fields from an invoice. Treating "best" as a single axis is the most common mistake in model selection, and it is why leaderboards that crown one winner age badly within weeks.

There is a third, quieter reason: the frontier moves monthly. Between spring and midsummer of 2026 alone, OpenAI shipped the GPT-5.6 family, Anthropic pushed out Opus 4.8 and the Mythos tier, Google refreshed Gemini, and open-weight labs released DeepSeek V4, GLM-5.2, Qwen3-Coder, and Kimi K3 in quick succession. If your codebase hard-codes one vendor, every one of those releases is a migration you cannot take advantage of. A routing layer turns them into config changes instead.

The task map, as of August 2026

Here is how we split a development workflow into task categories, with the models we reach for in each. Treat the names as a dated snapshot. The tiers are what matter, because the tiers are stable even when the occupants change.

A few notes on the reasoning behind the tiers.

For the hardest work, correctness matters more than latency or price, so you spend up. Opus 4.8 and GPT-5.6 Sol trade the top spot depending on which coding and reasoning benchmark you weight, and both sit near the ceiling on the current intelligence indexes. Gemini's Pro line leads the abstract and scientific reasoning tests. If you are running an agent on a genuinely long task where a wrong turn early costs an hour, this is the tier that earns its price.

The everyday tier is where most of your engineering volume should live. Sonnet 5 is the default in Anthropic's own apps for a reason: it clears the bar for real feature work at a fraction of frontier cost. GPT-5.5 plays the same role in the OpenAI ecosystem. If you standardize your default driver here and only escalate when a task actually needs it, you have already captured most of the savings orchestration offers.

The high-frequency tier is about running the same call thousands of times. Haiku 4.5 is priced to be queried constantly, around one dollar in and five out per million tokens, which is exactly what you want inside a tight tool loop. Gemini's Flash line is the natural runner-up, fast and cheap with a long context window when you need it. The trick is that these models are good enough for the small, well-scoped calls that make up most of an agent's inner loop, and you save the expensive model for the moment the loop actually gets stuck.

Bulk deterministic work is the most underrated place to save money. Generating commit messages, writing docstrings, tagging tickets, and summarizing diffs do not need a frontier brain. Point them at the cheap tier or an open-weight model and the quality is fine. Teams that skip this step pay frontier prices for work a small model does just as well.

Open-weight models deserve their own line because the reason to reach for them is usually not raw quality. DeepSeek V4, GLM-5.2, and Qwen3-Coder now land within a few points of the proprietary frontier on the coding and reasoning benchmarks, and you can run them on your own hardware with no per-token pricing and no data leaving your walls. That last part is the real draw for anything sensitive. The catch is that open-weight models shine when you control the runtime: enforced diff output, automated test runs, a repeatable evaluation harness. Drop them into an unstructured chat and they underperform. Give them a tight harness and they are a genuine cost advantage.

A route through one feature

Abstract tiers are easier to see against a real path. Take a single feature request moving through a team that routes by task.

Planning comes first. A developer describes the change and the desired behavior, and that goes to the reasoning tier. Opus 4.8 or GPT-5.6 Sol reads the relevant parts of the codebase, proposes an approach, and flags the edge cases. This is the one call where you happily spend, because a good plan saves a dozen bad iterations later.

Scaffolding and the main implementation drop to the everyday tier. Sonnet 5 takes the agreed plan and writes the feature, following the repository's conventions. Most of the token volume for this feature lives here, at a price you can run all day.

Boilerplate and tests move to the high-frequency tier. Haiku or Gemini Flash generates the repetitive test cases, fills in the obvious branches, and handles the mechanical parts of the diff inside a loop, escalating back up only when a test refuses to pass and something actually needs thinking through.

The mechanical wrap-up, commit messages, a changelog entry, updated docstrings, runs on the cheap or open-weight tier. None of it needs reasoning.

Review closes the loop back at the reasoning tier. A frontier model reads the final diff for the things a fast model misses: a subtle N+1 query, an injection path, a missing edge case. And if the feature involves pulling requirements out of a long spec or a stack of PDFs, that retrieval goes to the long-context tier, usually Gemini, in parallel.

One feature, five or six different models, each doing the part it is best and cheapest at. That is orchestration in practice, and no single model on the market would have been the right call for every hop.

The plumbing

You do not want this routing logic scattered through your application code. It belongs in a gateway, a layer that sits between your app and the providers and speaks one interface to all of them.

Three tools dominate this space right now. LiteLLM is the self-hosted choice: you run it in your own environment, it exposes 100-plus providers behind an OpenAI-compatible API, and it gives you budgets, virtual keys, and native hooks into observability tools like Langfuse. It trades some operational ownership for full control, which suits teams with platform engineers. OpenRouter is the managed opposite: a hosted SaaS gateway with unified billing and a huge model catalog, no infrastructure to run, ideal when time to market beats deep customization. Portkey sits in the enterprise slot, with its own observability dashboard, caching, and guardrails built in. Many teams pair them, running LiteLLM for control and reaching through OpenRouter for breadth.

Above the gateway sits the application layer, where LangGraph, LlamaIndex, and the Vercel AI SDK handle prompt orchestration, tool calling, and streaming. These are complementary layers, not competitors. The gateway decides which model gets the call; the SDK decides what the call is.

For the cheap and private tier, put a local serving stack behind the router. vLLM, Ollama, or llama.cpp host your open-weight models, and a router that spans both hosted and local pools sends the sensitive or high-volume traffic to your own hardware while frontier work still flows out to the APIs.

The routing itself takes a few forms worth naming. Task-tiered routing keys off the kind of work, which is what the map above describes. Difficulty-based routing puts a small, cheap classifier in front that reads the request and decides whether it needs the frontier or the fast tier, so easy questions never reach the expensive model. Fallback chains reroute automatically when a provider returns an error or goes down, which keeps you up during an outage. And prompt caching is the cost lever people forget: when large parts of your prompt repeat, caching them at the gateway cuts the bill on every hit.

What teams get wrong

A handful of mistakes show up again and again.

Routing on brand instead of task shape is the first. "We use Claude" or "we're an OpenAI shop" is a procurement decision, not a routing strategy. The unit of routing is the task, not the vendor.

Skipping the eval harness is the expensive one. Without a way to measure whether the cheap model is actually good enough for a given task on your own traffic, you are guessing, and you will either overspend out of caution or ship regressions out of optimism. Run a two-week pilot on real traffic before you commit a route, and keep the harness running so you can re-check when models change.

Chasing the top of the leaderboard for work that does not need it is a slower kind of waste. The model that wins the hardest reasoning benchmark is rarely the one you want writing your docstrings, and paying frontier prices for cheap-tier work adds up across a month.

Ignoring caching and fallbacks leaves money and uptime on the table. Both are close to free to set up at the gateway and both pay for themselves quickly.

The move that lasts

Every model name in this post will look dated by the fourth quarter. Half of the launches happened in the five weeks before it was written, and the release cadence is not slowing. That is exactly why the strategy matters more than the roster.

Classify your tasks. Map each class to a tier. Put a gateway in front so switching a model is a config change, not a rewrite. Measure each route with a real eval harness on your own traffic. Then re-check the mapping every month, because the occupant of each tier will keep changing while the tiers themselves stay put.

Get that structure right and the monthly churn stops being a problem you chase and becomes an advantage you compound. When a new model beats the current pick on your evals, you promote it in one line and move on. That is the whole point of orchestration: you stop betting the workflow on any single model, and you let each piece of work find the model that suits it.

Building AI systems and want a second opinion on how to route them? That is the kind of work we do at Yatzon.