AI shopping assistants flunked 43% of Visa's buying tests

AI shopping assistants got it wrong in more than four of every 10 buying tests run by Visa. Worse, in 54% of those failures the AI still went ahead with a purchase. It either picked a product that did not fit the shopper or missed some of what the shopper needed.

What Visa tested

Visa ran 22 AI models from 12 labs through 15 simulated shopping challenges. They included buying televisions, laptops and personal care appliances. The AI had to talk with a pretend customer, work out what that person needed, and then choose a suitable product. If nothing on offer fit, it was supposed to decline to buy.

Across all attempts, the models passed 57.2% of the time. Results varied a lot by model. The best passed 82.2% of tests. The weakest passed 28.9%.

Results were not steady

Each model tried every challenge three times. All three tries succeeded in only 34.2% of model and scenario pairings. In plain terms, the same AI could get the same task right once and wrong the next time.

Asking good questions seemed to matter. The five models best at uncovering a customer's unstated needs passed 70.2% of attempts. The five weakest at it passed 44.4%.

Shoppers are wary

Visa also pointed to a survey it released in September. Of U.S. consumers, 72% said they had used an AI assistant. Only 23% trusted generative AI to handle payments for them.

In the same research, 61% of respondents trusted Visa's brand to handle AI-started payments. The survey, run by Harris Poll, covered 2,065 U.S. adults on May 26 to 28. The payment questions went to about 1,030 people per brand. Visa is building its own technology for consumer-approved AI purchases, so it has a stake in this topic.

What to do

• Check what an AI assistant picked before any money moves.
• Say your needs clearly, including budget, size and must-have features.
• Be cautious about letting an assistant complete a purchase on its own.

What happens next

This first test did not cover checkout, payments or order placement. Visa plans to add more products and shopping scenarios, and eventually to test those steps. Visa says trust will be key to getting people to use AI for buying.