A tougher AI test shows free models trail the best paid ones by more

A tougher AI test has reshuffled the rankings, and open models, whose inner workings are shared publicly, now look further behind the best closed ones. Artificial Analysis, which ranks AI models, overhauled its Intelligence Index. It now puts the gap between open models and the top spot at 12 points, up from 3.

What changed

The biggest shift is how the test works. For the first time, 60 percent of the index uses private questions, so models cannot be trained on them in advance. Doing that with known questions is called "benchmaxxing." Kimi K3.1 Pro, an open model, launched in fourth place. Under version 5 it falls to ninth, behind eight closed models. Grok 4.7 also lost 12 points.

AI researcher Nathan Lambert says benchmaxxing is only part of the story. Many training gains now come from practice tasks that resemble the benchmarks. Keeping up across many areas at once takes complex training setups, and not every lab has them.

What is still unclear

Private tests have a cost. Outsiders can no longer check the questions for mistakes. Artificial Analysis had to revise ambiguous and flawed questions in its AA-Omniscience knowledge test for the new index. The material does not say which models now lead, or how the scores match everyday business work. Old rankings may have flattered some models, so test any tool on your own real tasks before you commit.