Why a cheaper model can cost more halfway through a task
The price list says Model B is cheaper than Model A. So it seems obvious that moving a task to B halfway through will save money. In long agent sessions that is often wrong, and the reason is worth understanding if you plan to route between models.
Agents re-read a lot
An agent working through a task keeps adding to a growing context: the instructions, the files it has read, the tool results, and its own earlier steps. On every new step, the model reads that whole context again before producing the next move.
Many providers make this cheaper with prompt caching. If the start of the context is identical to what was just processed, the provider can reuse that work and charge less for it. Terms and time limits differ between providers, so check yours. The effect is the key idea: staying with one model keeps the cache warm.
What switching does
A cache belongs to a model. When a session moves to a different model, the new model has not seen that context, so the whole thing is processed again at the ordinary input rate. A write-up of this problem puts it plainly: in a long session, a cheaper model can still end up costing more because switching forces the accumulated context to be paid for as fresh input.
Figure 1 · Where the cost comes back
Stay on one model
Switch at step 3
What the field is learning
In August 2026, Microsoft Azure Research published what a secondary write-up describes as the first production-scale study of agentic coding workloads: one week of sampled GitHub Copilot traffic, covering 13.5 million sessions and 760 million model calls. According to that write-up, routing at the boundaries between phases of an agent’s work did much better, because the harness knows when one stage ends and the next begins. We could not open the full write-up in our review, so read the original before relying on the details.
LangChain’s router, covered in our other post, takes a related approach. It decides once, on the first message in a thread, and keeps that model for the whole thread.
Practical rules
- Route at natural boundaries. New task, new thread, or a clear change of phase, such as planning to writing.
- Avoid ping-pong. Do not alternate between models on consecutive steps.
- Measure total cost per finished task. Price per token alone hides the cost of re-reading.
- Compress before switching. A short summary passed to the new model can be cheaper than the full history.
The lesson is that routing is a state decision as much as a price decision, and the harness is where that state lives. It is one reason we are building Router and Harness to work together.