A Chinese AI model fell from 4th to 9th after a tougher test

A Chinese AI model that ranked fourth best in the world last week now ranks ninth. Kimi K3.1 Pro, from Beijing-based Moonshot AI, slid after Artificial Analysis updated its widely watched ranking with a tougher test that keeps most of its questions secret.

What happened

Kimi K3.1 Pro debuted to a lot of fanfare. It took fourth place on Artificial Analysis' Intelligence Index v4. It also took the top spot on Terminal Bench 4.0, which focuses on computer use. Then, on Tuesday, Artificial Analysis announced Intelligence Index v5, which it called its biggest upgrade since v4 launched in January. Kimi dropped to ninth.

Why the test changed

About 60% of the new test uses private questions, which Artificial Analysis calls held-out answers. The goal is to curb what the industry calls benchmaxxing. That means preparing a model for a test by feeding it old questions that are already posted online, in places like GitHub and Hugging Face.

Think of a student who memorized last year's exam. A new exam with hidden questions shows who actually learned the material.

The open-weight angle

Moonshot has not yet released Kimi's weights, and will not until November 4. Open weights means a model's inner settings are shared publicly, so others can use and adapt it. The Decoder had noted that a release would have made Kimi the top open-weight model by a wide margin, shrinking the gap with closed models from 12 points to 3. Under v5, Artificial Analysis says the gap widens from 3 points back to 12.

Who is on top now

Google's Gemini 4 Carbon is the new number one on v5, with a score of 59. It was evaluated but not released. Google featured the ranking in the second sentence of its announcement on Wednesday. Carbon is also second only to Claude Opus 5.5 on Artificial Analysis' Coding Agent Index.

There may be other reasons Chinese models dropped. Physicist Stephen Hsu wrote on X that U.S. models are ahead in math and science. He said Chinese labs he has talked to have not prioritized that area.

What to do

Artificial Analysis is a private firm that appears to have been founded in 2023. It is not a standards body or an academic institution, yet its rankings carry real weight. Before choosing an AI tool for your business, treat any best-model claim with care.

• Check when the ranking was made and which version of the test was used.
• Look for tests that keep some questions private.
• Try the tool on your own real tasks before you commit.