Three Claude models, 20 Shopify products, 1,980 picks, no agreement: the data, charted

On October 13, Shopify opened Shop MCP, a single connection that lets any AI agent search Shopify's catalog of more than a billion products, build carts and complete Shop Pay checkout. Searching needs no API key. That makes the catalog a shelf any agent can shop from.

A catalog returns a list. Something still has to choose one product from it, and inside an agent that something is a language model. Slobodan Manic, who runs the No Hacks consultancy, set out to measure that step on its own. His write-up, published on Search Engine Journal on November 2, is the most detailed public test we have seen of how a model picks from a product list. We charted the parts that matter most for consumer brands.

How the test worked

Manic ran each search once through Shop MCP as an anonymous US buyer and saved the top 20 results. Eight skincare searches were fetched on October 15, and six more (backpack, coffee beans, running shoes, water bottle, wireless earbuds and yoga mat) on October 24.

Each model saw only what the catalog returned for each product: title, store, price, star rating and review count, a one-line summary and the first three features. The instruction was short: you are a shopping agent, choose the one product you would buy, reply with the pick and a reason. The shopper's request was just the search words.

The models were Claude Haiku 4.5, Sonnet 5 and Opus 5, run with default settings, no tools and no web access. Every pick was a fresh call.

Run set	What changed	Picks per model	Total picks
Six product searches	Nothing, search words only	50 per search	900
Sunscreen, asked two ways	Plain request vs "under $25, for oily skin"	100 per version	600
Eight skincare searches	Catalog order vs reversed order	10 per version per search	480

Source: Slobodan Manic via Search Engine Journal.

Finding 1: each model has a favorite, and they don't share

On the six product searches, the three models' most-picked product matched in none. Haiku and Sonnet agreed on three, Sonnet and Opus on two, Haiku and Opus on one.

Each model was very sure of itself, though.

Show the data

Model	Value
Sonnet 5	94%
Opus 5	89%
Haiku 4.5	67%

Source: Slobodan Manic, via Search Engine Journal (Nov 2, 2026). 50 picks per model per search. The three favorites matched in none of the six searches.

On three of the six searches the different favorite also came from a different store. On water bottle, Haiku and Sonnet mostly picked nobl's bottle and Opus always picked Owala's. On yoga mat, Haiku picked POPFLEX and the other two picked Liforme.

Shopify's own ranking counted for less than you might expect. Haiku picked the catalog's top product 8% of the time, Sonnet never, Opus 20%.

On one model, one product gets nearly every pick. Change the model and a different product does.

Finding 2: review counts drove two of the three models

For backpack, coffee beans and water bottle, the product Haiku and Sonnet picked most was the one with the most reviews in the list. Their reasons often opened with the count.

The water bottle shows what that costs. Shopify's catalog listed nobl's bottle at 5.0 stars from 447,221 reviews. Haiku picked it 45 times out of 50 and Sonnet 39 times. When Manic checked nobl's own product page on October 24, it showed 5.0 from 6,537 reviews, and the catalog record for one color showed 4.2 from 1,674. Manic could not say what the 447,221 counts. It is about 68 times the number on the brand's page, and the models had no way to check it.

Opus never picked that bottle. It picked Owala's bottle (4.6 from 34,394 reviews) all 50 times.

Finding 3: Opus buys for a shopper nobody described

Opus explained itself in a way the others did not. In 222 of its 300 picks on the six searches, its reason called the choice "safe" or "safest." Haiku did that once and Sonnet 11 times. Given only "coffee beans," Opus picked the medium roast 48 times out of 50, calling it "the safest crowd-pleaser."

Manic's reading is that every model has a default when the shopper says little. Opus states its default. Haiku and Sonnet default to review counts.

Finding 4: list position moved picks, and no reason admitted it

In the skincare runs, Manic showed each model the same 20 products in the catalog's order and then reversed, so the top product went from first to last.

Model	Listed first	Listed last
Haiku 4.5	28	11
Sonnet 5	15	11
Opus 5	24	10

Source: Slobodan Manic via Search Engine Journal. Eight skincare searches, 10 picks per model per version.

The effect was strong on some searches (mineral sunscreen, lip balm) and absent on most. The striking part: of 480 reasons, none mentioned where a product sat in the list. A shopper reading the agent's explanation would never know.

Finding 5: the shopper's words beat the catalog's ranking

The most useful result for brands came from one change to the request. Asked for "mineral sunscreen SPF 50," the models often picked the catalog's top product, a $48 sunscreen. Asked for "mineral sunscreen SPF 50, under $25, for oily skin," none of them did.

Show the data

Model and request	Value
Haiku, plain request	100
Sonnet, plain request	90
Opus, plain request	33
Any model, under $25 and oily skin	0

Source: Slobodan Manic, via Search Engine Journal (Nov 2, 2026). With the budget and skin type added, 298 of 300 picks were under $25.

Among the six sunscreens under $25, Sonnet picked ATTITUDE's ($22.95, 4.7 from 292 reviews) every time. Opus split between ATTITUDE (67 picks) and Sun Bum's lotion (33). The Sun Bum product had the lowest rating of the six, 4.2 from 105 reviews, but its listing was the only one that said "without clogging pores." Opus cited that phrase in 31 of its 33 reasons for choosing it. Badger's sunscreen had the best rating of the six, 4.9, and was never picked; its listing talked about "organic plant oils."

The models also repeated claims straight from listings. "Clinically proven" appeared in 63 reasons, and 61 of those came from the chosen product's own listing. All 22 reasons saying a retinal product works "11x faster" than retinol came from listings. The model cannot check a claim like that. The shopper hears it from the agent.

What this means for a CPG brand

The test is narrow: three Anthropic models, a bare instruction, no shopping app around them and nothing bought. Manic says so plainly. Real agents add their own instructions, which are private and differ between apps. But the layer he tested sits under all of them, and four lessons carry over.

Check what the catalog says about you

Models reason from the catalog record, not your product page. Search for your own products the way an agent would and read the rating, review count, summary and first features. If a number is wrong or inflated, the water bottle case shows it can swing picks heavily, in either direction. Shopify's Muse announcement this week makes the same point from the merchant side: agents can only recommend what they understand, and catalog data is what they read.

Put the shopper's need in plain words

"Without clogging pores" won a third of Opus's oily-skin picks for a product with the lowest rating in the group. Write the phrases your customers use when they describe their problem: skin type, diet, allergen, fit, use case. Our CartTrace analysis found agents fail most often at reading product facts, which points the same way.

Keep every claim true and supportable

An agent repeats your claims as its own reasons. That raises the stakes on substantiation; we look at this in today's think piece on listings as agent speech.

Test your category across models

There is no single ranking to monitor, a point we made in stop asking where you rank. What you can do is run your top searches through several models with competitors in the list and see who gets picked. We wrote up a repeatable method in today's playbook. If you would rather not build it yourself, XENA Intelligence runs shopper-style questions across assistants, and its SearchPanel tool tracks how individual products come out.

Limits worth repeating

• Only Claude models were tested. Other companies' models may pick differently.
• No app instructions, no shopper history, and in most runs no stated need.
• Lists were fixed snapshots from October 15 and 24 for an anonymous US buyer.
• "Safe" was counted by searching the reasons for "safe" or "safest."
• Nobody bought anything.

Even so, it is the clearest public evidence yet that in agent shopping, the model is a second ranking layer you cannot see. The parts you control are the facts and words it reads.