Most of what we know about AI chatbots goes out of date within about a year

Much of what people have read about AI chatbots is already out of date. Stanford researchers Owen Queen and James Zou tested 50 widely cited claims about large language models, the technology behind chatbots, against 20 current models. They found that 44 percent of the claims have "expired." For the 22 findings with an estimated expiration date, the average lifespan was 15.7 months.

What the study found

The models were released between July 2024 and September 2026 by eight different developers. For the expired claims, the original effect has largely disappeared. Another 34 percent have weakened significantly. Only 16 percent still hold, and 6 percent have gotten stronger. After one year, 16 percent of findings had expired. After two years, that figure reached 39 percent.

What fades fastest

Claims about what chatbots can't do go stale first. Five of nine reasoning-related findings are considered expired, and none have held up or grown stronger. They include counting letters in "strawberry," arithmetic in unfamiliar number systems, losing information in the middle of long text, and giving up correct answers when a user pushes back. The researchers suspect these errors have a clear right answer, which gives developers a target to train toward.

Habits and tendencies last longer. Of the 11 findings that held or strengthened, three involve models acting as judges. They prefer their own answers, rate accepted and rejected papers similarly, and are swayed by spin in medical abstracts. Three more involve sycophancy, meaning agreeing with the user. A left-leaning political bias, near-identical answers to open-ended questions, and uneven random numbers also persist.

Fake sources: better, but not everywhere

The team revisited a 2023 study by Walters and Wilder, in which GPT-3.5 made up 55 percent of its scientific references. With the same prompts, GPT-6.1 Sol made up 0 percent and Claude Sonnet 5.5 made up 1 percent. But Qwen3.7-Flash, also a 2026 model, still invented 69 percent. The authors warn against reading this as an all-clear. On two-step questions that combine two known facts, Claude Sonnet 5.5 still fails about 10 percent of the time, down from 43 percent reported for ChatGPT.

What to do

Results differ a lot between models, even from the same period. Among models released from July to September 2026, expired findings ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Release date, provider and model size did not reliably predict the pattern. In practice:

• Treat older tips about what AI can't do as possibly outdated.
• Test the specific tool and version you use today.
• Check answers and sources on anything that matters, such as money, health or legal questions.

What happens next

Queen and Zou say findings about chatbots should be presented as observations about specific models at a specific time, not as general truths. They want papers to record the model, version and date, and to publish prompts and code so others can re-test on newer models.