Visa graded 22 AI shopping assistants: 57% of runs passed, and half the failures still bought something

Most of what has been published about AI shopping agents measures whether they can finish a checkout. Visa measured something earlier and, for brands, more important: whether the assistant works out what the shopper needs before it picks a product. The answer, across 22 models, is a little better than a coin flip.

Visa released its Agentic Commerce Benchmark on November 5, 2026, with a press release and a much more detailed results page on Visa Payments Frontier. We went through both. Below are the numbers that matter for a packaged goods brand, what the test can and cannot tell you, and what we would change on a product page because of it.

How the test works

Each scenario, which Visa calls an episode, is a conversation between two AI models. One plays the customer. The other, the model being tested, plays the shopping assistant. The customer opens with a few stated needs and holds the rest back. A hidden fact only comes out if the assistant asks a relevant question or takes an action that would reveal it. The assistant searches a simulated market that is identical for every model, then recommends something. The customer buys it or declines.

A run passes only if the decision is right: the product meets every requirement in an answer key no model sees, and each requirement was actually found during the conversation. If nothing on the market fits, declining is the right answer. A lucky pick that happens to fit does not count.

The public example shows why this is hard. A shopper named Sam needs a 65-inch TV with at least 30-day returns. Only by asking does the assistant learn the budget is up to $1,200 all in. Only by asking Sam to check with the family does it learn the spouse insists on Dolby Vision, the kid's phone records HDR10+ video and the console wants native 120 Hz. The market starts with 265 TVs; after the stated needs there are 65 left, and after every hidden one, 20. On that episode, 62% of the models' runs passed.

The first round covered 22 models from 12 labs, 15 episodes and three runs per episode per model. Visa lists TVs, laptops and personal care appliances among the categories. Fourteen of the episodes are kept private so the test cannot be gamed.

How the runs ended

Show the data

Outcome	Share of runs
Passed	57.2%
Purchase did not pass	23.1%
Other outcomes	19.7%

Source: Visa Payments Frontier, Agentic Commerce Benchmark. Other outcomes: the customer declined though a fit existed, usually because the assistant said there was none, or the run ended with no decision (tool-call limit or reply format).

Visa's press release puts the failure rate at 42.8%. The part we would underline is the middle bar. In almost a quarter of all runs, the customer bought something that did not pass. Visa's own framing is that, of the runs that failed, 54.0% ended with the customer buying an offer that missed a requirement, or one chosen before a requirement came out.

The failures were not spread evenly. According to the results page, the OpenAI models ended the most runs with no purchase even though a fit existed (GPT-6.1 Terra in 38% of its runs), while some other models leaned the other way; Gemini 3.1 Pro made a purchase that did not pass in 51% of its runs. Two models with the same pass rate can fail in opposite directions: one buys the wrong thing, the other buys nothing.

Asking was the skill that mattered

Visa reports two comparisons between groups of five models. Both are in the chart below.

Show the data

Group of five models	Pass rate
Uncovered the most hidden buying factors	70.2%
Highest cost per run	67.1%
Lowest cost per run	50.2%
Uncovered the fewest hidden buying factors	44.4%

Source: Visa press release, November 5, 2026. Each figure is the average of five models. Groups overlap and are not independent.

The gap between the best and worst askers, 70.2% against 44.4%, is bigger than the gap between the most and least expensive models, 67.1% against 50.2%. Spending more on the model helped. Asking better questions helped more.

The results page explains where failures come from: "capable models seldom misinterpret a fact once it's in their context. Their failures occur earlier." Visa lists five failure patterns the episodes are built to catch. Two of them are about product data, and they are the ones we would print out:

• Satisfying every stated requirement while never asking about unstated ones.
• Treating a value missing from a listing as acceptable.

The other three are asserting nothing fits while a fitting offer sits unexamined in earlier results, talking the customer out of a correct choice over a concern no requirement supports, and agreeing on a bundle in conversation, then finalizing a selection without it.

Passing once is not passing reliably

Measure	Share
A single run passed	57%
Two runs of the same episode both passed	42%
All three runs passed	34%
Best model, share of episodes with all three runs passed	67% or less

Source: Visa Payments Frontier. The press release gives the all-three figure as 34.2%. No model passed all three runs on more than 67% of the episodes.

For brands this is the number to sit with. If you test an assistant once with a shopping question about your category and it picks you, that tells you very little. The same model, asked the same thing again, may land somewhere else. We made a similar point about Claude models disagreeing on Shopify picks; Visa's data puts a rate on it.

The best performers, Gemini 4 Argon and Muse Wonder, each passed 82.2% of their runs. Visa notes that each rate has a wide 95% confidence interval and that the next eleven models overlap them, so it says no model clearly leads.

What the test does not cover

Be careful about stretching this to grocery and household goods. The public scenario is a TV, and the listed categories are electronics and a personal care appliance. Requirements for a laundry detergent or a snack are fewer and simpler than for a TV, so pass rates could be higher. They could also be lower, because the deal-breakers in food (an allergen, a certification, a pack size) are exactly the kind of thing a shopper leaves unsaid.

It is also a simulation. Both sides are AI models, the market is fixed, and the test stops at the decision. Visa says checkout, payment and order placement come next. And it is a vendor's benchmark: Visa runs a portfolio of agentic commerce products, though it says the benchmark is separate research and that it has shared it with model providers.

Finally, it measures models, not shopping apps. ChatGPT, Gemini, Alexa for Shopping and the rest wrap their models in product data, retrieval and interfaces that may help or hurt.

What it means for a CPG brand

Your page has to answer the question the assistant forgot to ask. If the common failure is not asking about unstated requirements, and treating a blank field as acceptable, then the brand that writes its deal-breakers down plainly is protected against both. Free from the top nine allergens, unscented, safe for septic systems, fits a standard dispenser, count per pack: if those facts are on the page and in the feed, an assistant that skips the question can still get it right. We wrote a step-by-step playbook for that today.

A wrong purchase is your problem even when the agent made it. When an assistant buys your product for someone it does not suit, the review, the return and the lost repeat purchase are yours. Tip 25 covers what to do when an assistant gets your product wrong, and General Machines' simulated store visits, which we charted in our CartTrace post, found the same pattern: agents rarely broke at checkout and often broke at reading the product.

Do not judge your AI visibility from one run. With all three runs passing only about a third of the time, a single check is noise. Ask the same set of questions on a schedule, more than once per assistant, and look at the share. That is how teams that do this seriously work, whether in a spreadsheet or in a tool like SearchPanel, which XENA Intelligence makes for scoring each product across assistants.

Trust is still the bottleneck, and accuracy feeds it. Visa's own Trust Index, released in September, found that 72% of US consumers had used an AI assistant but only 23% trusted generative AI to handle payments on their behalf, according to its release. Benchmarks like this one show why the gap exists. They also show where brands can help close it: by publishing facts complete enough that a hurried assistant has less room to guess. If you are deciding whether to let agents buy from your own store at all, tip 33 and tip 46 cover the first steps.

The headline number, 57.2%, will be quoted as proof that agents are not ready. We read it differently. The models that asked more questions did much better, which means the failures are mostly about missing information. Some of that information lives with the shopper. A lot of it lives on your product page.