Visa tested 22 AI shoppers, and most still make costly mistakes

Visa has built a test for AI shopping assistants, and the first results are not flattering. Across 22 AI models, only 57.2% of realistic shopping tests were passed. In more than half of the failed runs, the AI still went ahead and bought something that did not fit the customer's needs.

How the test worked

The Visa Agentic Commerce Benchmark tested 22 models from 12 AI labs on 15 shopping tasks, such as buying a TV, a laptop or a personal care appliance. The AI had to talk with a shopper and work out what they needed, because people often do not know all their requirements up front. If no product fit, the AI was supposed to decline to buy.

What the results show

• The best models passed 82.2% of attempts. The weakest passed 28.9%.
• Each model had three tries per task. All three passed only 34.2% of the time.
• In 54% of failed runs, the AI still made a purchase.
• The five most expensive models passed 67.1%, against 50.2% for the five cheapest.

Asking good questions mattered. The five models that uncovered the most hidden requirements averaged a 70.2% pass rate. The five that uncovered the fewest averaged 44.4%. In one TV task, the AI had to find out the budget, format compatibility, HDR10+ support and gaming needs.

Why it matters

The results show that an AI can sound confident and still miss what you need. It may also give a different answer when asked to repeat the same job. Shoppers seem aware of this. Visa's Trust Index, released in September 2026, found that 72% of US consumers had used an AI assistant, but only 23% trusted AI to handle payments for them. A SEON survey found half of surveyed US adults uncomfortable with AI agents using their card details.

What to do

If you or your business use an AI assistant to pick or buy products, treat its choice as a suggestion. Check the price, the specifications and anything that is a must-have before you pay. The test suggests the weak spot is the AI not asking enough questions, so state your requirements clearly at the start.

What happens next

Visa plans to add more shopping scenarios and product categories. It also wants to extend testing to checkout, payment and order placement. The company has shared the benchmark with model providers for feedback and published a whitepaper on the research. Rubail Birwadker, Global Head of Growth at Visa, said trust is key to making AI-powered commerce work. The material does not say which models scored best or worst.