Why a cheaper model can cost more halfway through a task

    The price list says Model B is cheaper than Model A. So it seems obvious that moving a task to B halfway through will save money. In long agent sessions that is often wrong, and the reason is worth understanding if you plan to route between models.

    Agents re-read a lot

    An agent working through a task keeps adding to a growing context: the instructions, the files it has read, the tool results, and its own earlier steps. On every new step, the model reads that whole context again before producing the next move.

    Many providers make this cheaper with prompt caching. If the start of the context is identical to what was just processed, the provider can reuse that work and charge less for it. Terms and time limits differ between providers, so check yours. The effect is the key idea: staying with one model keeps the cache warm.

    What switching does

    A cache belongs to a model. When a session moves to a different model, the new model has not seen that context, so the whole thing is processed again at the ordinary input rate. A write-up of this problem puts it plainly: in a long session, a cheaper model can still end up costing more because switching forces the accumulated context to be paid for as fresh input.

    Figure 1 · Where the cost comes back

    Stay on one model

    Step 1full readSteps 2–5mostly cached

    Switch at step 3

    Step 1full readStep 2cachedStep 3full re-read on new model
    Illustrative only, not measured. The point is where the expensive full read happens again.

    What the field is learning

    In August 2026, Microsoft Azure Research published what a secondary write-up describes as the first production-scale study of agentic coding workloads: one week of sampled GitHub Copilot traffic, covering 13.5 million sessions and 760 million model calls. According to that write-up, routing at the boundaries between phases of an agent’s work did much better, because the harness knows when one stage ends and the next begins. We could not open the full write-up in our review, so read the original before relying on the details.

    LangChain’s router, covered in our other post, takes a related approach. It decides once, on the first message in a thread, and keeps that model for the whole thread.

    Practical rules

    1. Route at natural boundaries. New task, new thread, or a clear change of phase, such as planning to writing.
    2. Avoid ping-pong. Do not alternate between models on consecutive steps.
    3. Measure total cost per finished task. Price per token alone hides the cost of re-reading.
    4. Compress before switching. A short summary passed to the new model can be cheaper than the full history.

    The lesson is that routing is a state decision as much as a price decision, and the harness is where that state lives. It is one reason we are building Router and Harness to work together.

    Sources

    Want help putting this into practice?

    Book a free consultation and tell us about your team or school.

    Book a free consultation