A new model every week: how to decide whether to switch
November 2026 has been busy. One tracker lists 17 new models from nine providers so far this month, including Qwen4 Max on 16 November, GPT-6.2 Astra on 17 November, and Gemini 4 Neon on 18 November, after Claude Fable 5.2 on 9 November. The list includes image and speech models as well as language models, but the pace is clear.
Launch posts report results on benchmarks someone else chose. They cannot tell you whether a model is better at your tasks. You can find that out in an afternoon.
A six-step test
Figure 1 · The test
Collect real tasksDefine a passRun both modelsScore blindDecide per taskRe-run on change
- Collect 20 to 30 real tasks. Use actual inputs from your work, including messy and awkward ones.
- Write what a pass looks like. Be specific. “Correct total and cites the right clause” beats “good answer”.
- Run the current and the new model on the same tasks. Keep prompts identical at first.
- Score without knowing which is which. Hide the model names, so a famous name does not sway the score.
- Record quality, cost, and speed together. A small quality gain may not justify double the cost, and the reverse can also be true.
- Decide per task type, not globally. The new model may win on some work and lose on the rest. That is where routing helps.
A scorecard you can copy
Figure 2 · Example scorecard
| Task | Current model | New model | Cost | Time | Notes |
|---|---|---|---|---|---|
| Summarise a contract | Pass | Pass | New: higher | New: faster | Both cite the right clause |
| Extract invoice totals | Fail | Pass | Similar | Similar | Current mis-reads a table |
| Draft a polite refusal | Pass | Pass | New: lower | Similar | Tone slightly better on new |
Traps to avoid
- Testing on easy tasks. Include the cases that currently fail.
- Changing the prompt for one model only. Do that in a second round, and note it.
- Trusting one run. Outputs vary. Run hard cases more than once.
- Forgetting the data rules. A better model is no use if your policies forbid sending that data to it.
Keep the task set. Next month it will save you the first two steps.