Visa has launched the Visa Agentic Commerce Benchmark, finding that 22 AI models passed 57.2% of realistic shopping tests overall.
The benchmark requires AI models to identify a consumer’s needs through conversation, since shoppers often do not know all their requirements at the outset, such as wanting a TV without knowing the specifications they need. If no product meets the customer’s requirements, the AI should decide not to buy, and it must gather enough information to justify its decision rather than guess. The first round tested 22 models from 12 AI labs on 15 shopping tasks, including buying a TV, a laptop, and a personal care appliance.
Results show gaps in consistency
The best-performing models passed 82.2% of attempts, while the weakest passed 28.9%. The five models that uncovered the most hidden buying requirements averaged a 70.2% pass rate, compared with 44.4% for the five that uncovered the fewest. In one TV scenario, for example, the AI had to find out the customer’s budget, format compatibility needs, HDR10+ support, and gaming requirements. The five most expensive models per run passed 67.1% of the time, against 50.2% for the five cheapest.
Each model had three attempts at every scenario, and all three attempts passed only 34.2% of the time, with models that had already passed a scenario often failing when asked to repeat it. In 54% of failed runs, the AI still made a purchase, either recommending a product that did not meet the customer’s requirements or failing to uncover all of those requirements before buying.
Building trust in agentic commerce
Rubail Birwadker, Global Head of Growth at Visa, stated that AI agents could power the next era of commerce, but trust is key to making it work. According to Birwadker, the benchmark is part of Visa’s efforts to raise standards across the industry. Visa’s Trust Index, released in September 2026, found that 72% of US consumers had used an AI assistant, but only 23% trusted AI to handle payments on their behalf, a gap Visa aims to address through its Visa Intelligent Commerce programme.
Visa plans to add more shopping scenarios and product categories to the benchmark and to extend testing to checkout, payment, and order placement. It has shared the benchmark with model providers for feedback and published a whitepaper on the research. The findings come alongside other signs of consumer caution, including a SEON survey in which half of surveyed US adults said they were uncomfortable with AI agents using their card details.
