Hisham AlhussainNovember was ridiculous. GPT-5.1 on the 12th. Gemini 3 on the 18th. Claude Opus 4.5 on the 24th. Three frontier releases in under two weeks, each one topping some leaderboard, each with a launch thread full of charts.
I got the same question from three different people that month: "So which one should we use?" My honest answer was "I don't know yet, let's find out." That's not a dodge. It's the only answer that holds up.
Benchmarks aren't your workload
Public benchmarks are useful for a rough sense of capability. But your users aren't solving competition math or a standard coding benchmark. They're asking questions about your documents, in your domain, in your language, with your edge cases.
I've seen a model that's clearly "worse" on the leaderboards beat the leader on a specific task by a wide margin. Happens all the time.
Build a small eval set you trust
You don't need thousands of examples. Here's what I do:
- Collect 50 to 100 real examples. Pull them from logs or from users, and include the weird ones.
- Write down what "good" looks like. For some tasks it's an exact answer. For others, it's a checklist: did it cite the source, did it stay in scope, did it get the number right.
- Include the failures you care about. If hallucinating a policy would be a disaster, make sure your set tests for it.
- Grade with a mix. Code checks where you can, an LLM judge with a clear rubric where you can't, and a human spot check on a sample.
Look beyond accuracy
Once you've got scores, look at the stuff that actually decides whether a model works in production:
- Cost per task, not per token. A cheaper model that needs three retries isn't cheaper.
- Latency. Especially for anything user-facing.
- Consistency. Run the same eval a few times. Some models vary more than you'd expect.
- Tool use. For agents, how reliably it calls the right tool with the right arguments matters more than raw knowledge.
What I actually found
Without turning this into a leaderboard of my own: the three models had different strengths across my tasks, and the "best" one changed depending on what I was measuring. That's exactly why I'd never pick based on launch-day hype.
Make switching cheap
This ties back to something I wrote about in May. If your eval set runs with one command and your model is a config value, a month like November is fun instead of stressful. You run the evals, look at the numbers, and decide. Done.
If you take one thing from this post, let it be this: spend a day building your own eval set. It'll pay for itself the next time three models drop in the same week. And at this rate, that'll be soon.