Model routing belongs in the harness, not the gateway

    Most teams send every request to one model. Some add a gateway that picks a model by price. In October 2026, LangChain’s engineers argued that the choice should be made inside the agent harness instead, and they published results from an experiment on their own coding agent. Published results with a stated control are rare in this area, so they are worth reading closely.

    The argument

    A gateway sees a request. A harness sees the prompt, the tools, and the domain knowledge that were assembled for the task. The authors’ point is that judging how hard a task is needs exactly that context, which a generic gateway usually does not have.

    Figure 1 · Two places to route

    Gateway routing

    AppGatewaysees only the requestModels

    Harness routing

    TaskHarnessprompt, tools, contextRouterdecides hereModels
    Where the model choice is made. Simplified from the argument in LangChain’s post.

    What they tested

    They looked at a week of interactive threads from their coding agent and picked three models at different points on the cost and capability curve: a fast tier, a balanced tier, and a performance tier. A small router, built as middleware, ran once on the first human message in a thread and chose one tier for the whole thread.

    Then they ran an A/B test on 973 threads. Half were routed. The other half always used the top performance model. The main outcome was merged pull requests per thread, with thumbs-up and thumbs-down feedback as a second signal.

    Figure 2 · Cost per thread by tier

    Fast tier$0.097
    Balanced tier$1.50
    Performance tier$2.88
    Median cost per thread by tier, as reported. The spread between the cheapest and most expensive tier is about 30 times.

    What the numbers show

    Routed threads ended with a merged pull request 29.2% of the time, against 27.3% for the control. The authors report that difference as not statistically meaningful (p = 0.49), so they saw no measurable change in quality. Median cost per thread fell from $2.61 to $0.94, which is 64% lower. Routed threads went 56% to the balanced tier, 34% to the fast tier, and 10% to the performance tier.

    Figure 3 · Cost with and without routing

    Always the top model$2.61
    Routed by task$0.94
    Median cost per thread in the 973-thread test.

    What the numbers do not show

    • The control was the team’s most expensive model, so part of the saving is simply the cost of using it on easy work.
    • A second test, routing against always using the fast model, was stopped within a day because engineers reported poor output quality. Nothing can be concluded from it.
    • The task categories were heuristic, based on thread titles and metadata, and the authors call the router a proof of concept.

    That second point is the useful one. Moving everything to a cheaper model failed quickly. Choosing a model per task held quality in the first test. The skill is in the choosing.

    What we take from it

    1. Route on the task, not only on price. Ask what the work needs before asking what it costs.
    2. Keep the criteria readable. Rules in plain language can be reviewed and changed by the people who own the work.
    3. Measure outcomes. Cost per request is easy to track. Merged work, accepted drafts, and resolved tickets are the numbers that matter.
    4. Use your priciest model as the baseline. Then find the tasks where a cheaper tier is genuinely enough.

    This is the thinking behind Elsor Router and Elsor Harness, both in early access.

    Sources

    Want help putting this into practice?

    Book a free consultation and tell us about your team or school.

    Book a free consultation