A new model every week: how to decide whether to switch

    November 2026 has been busy. One tracker lists 17 new models from nine providers so far this month, including Qwen4 Max on 16 November, GPT-6.2 Astra on 17 November, and Gemini 4 Neon on 18 November, after Claude Fable 5.2 on 9 November. The list includes image and speech models as well as language models, but the pace is clear.

    Launch posts report results on benchmarks someone else chose. They cannot tell you whether a model is better at your tasks. You can find that out in an afternoon.

    A six-step test

    Figure 1 · The test

    Collect real tasksDefine a passRun both modelsScore blindDecide per taskRe-run on change
    The loop. The last step matters: a model update or a prompt change can shift results.
    1. Collect 20 to 30 real tasks. Use actual inputs from your work, including messy and awkward ones.
    2. Write what a pass looks like. Be specific. “Correct total and cites the right clause” beats “good answer”.
    3. Run the current and the new model on the same tasks. Keep prompts identical at first.
    4. Score without knowing which is which. Hide the model names, so a famous name does not sway the score.
    5. Record quality, cost, and speed together. A small quality gain may not justify double the cost, and the reverse can also be true.
    6. Decide per task type, not globally. The new model may win on some work and lose on the rest. That is where routing helps.

    A scorecard you can copy

    Figure 2 · Example scorecard

    TaskCurrent modelNew modelCostTimeNotes
    Summarise a contractPassPassNew: higherNew: fasterBoth cite the right clause
    Extract invoice totalsFailPassSimilarSimilarCurrent mis-reads a table
    Draft a polite refusalPassPassNew: lowerSimilarTone slightly better on new
    Illustrative layout only. The rows are examples, not results.

    Traps to avoid

    • Testing on easy tasks. Include the cases that currently fail.
    • Changing the prompt for one model only. Do that in a second round, and note it.
    • Trusting one run. Outputs vary. Run hard cases more than once.
    • Forgetting the data rules. A better model is no use if your policies forbid sending that data to it.

    Keep the task set. Next month it will save you the first two steps.

    Sources

    Want help putting this into practice?

    Book a free consultation and tell us about your team or school.

    Book a free consultation