How to run your own AI agent pick test for your product category, step by step

Brands have always been able to check where they rank on Google. Shopping agents took that away. Each agent asks a model to choose from a list, behind instructions nobody outside the company can read. As Slobodan Manic put it after testing three Claude models on Shopify's catalog: "With agents, everyone gets their own version of the decision, inside an app the store never sees, and there is nothing to monitor."

But part of the decision can be recreated. Manic's published test is a template, and its last recommendation to stores is simply: test your category. (We charted his results in today's data piece.) This playbook turns his method into steps a small brand team can run.

What you need

• A list of 3 to 6 shopper requests that matter for your category. Use the words customers use: "protein bar," "mineral sunscreen SPF 50," "kids' water bottle."
• A way to get the catalog results an agent would see. For Shopify's catalog, the Shop MCP changelog gives a command-line shortcut: install with npm install -g @shopify/shop, then run shop search "your request". Search needs no API key.
• Access to at least two AI models. Chat apps work for a small test; an API or a coding tool makes repeated runs much easier.
• A spreadsheet.

If you don't sell on Shopify, the method still works. Build the list from whatever an agent in your channel would see: a retailer's search results, a Google Shopping page, a marketplace listing grid.

Step 1: Freeze the list

Run each request once and save the top 20 results with the fields an agent receives: title, store, price, rating, review count, a one-line summary and the first few features. Note the date and whether you were signed in. Manic searched as an anonymous US buyer.

Freezing matters. If the list changes between runs, you can't tell whether a different pick came from the model or the catalog. Every run in your test should use exactly the same saved list.

While you are here, do the first and cheapest check: read your own products' records. Is the review count what your product page shows? Is the summary accurate? In Manic's test, one bottle's catalog record showed about 68 times as many reviews as the brand's own page, and two of the three models picked it heavily on that figure.

Step 2: Write one short instruction and keep it fixed

Use something close to Manic's: you are a shopping agent; a person asked you to buy something; here are the products the catalog returned; choose one and reply with the product number and a one-sentence reason. Ask for a fixed format so tallying is easy.

Then add the shopper request, which in the first run is only the search words.

Step 3: Run each model many times, fresh each time

Each pick should be a new conversation with no memory of earlier picks. One answer tells you almost nothing; you are looking for the model's habit.

Run	What changes	Suggested picks per model	Manic's run size
A. Baseline	Search words only	20 to 50 per request	50 per search
B. Shopper need	Add a real need, such as a budget, skin type or diet	20 to 50	100 per version
C. Position	Same list, reversed order	10 per version	10 per version per search

Source for Manic's run sizes: Slobodan Manic via Search Engine Journal. Suggested ranges are our recommendation for a smaller test.

If you are running by hand in a chat app, start smaller: 10 picks per model per request will still show whether a model has a strong favorite. Manic found each model's favorite product took between 67% and 94% of its picks on average, so a clear habit tends to show quickly.

Step 4: Add a real shopper need

Repeat run A with one thing added that real customers say: "under $25," "for oily skin," "gluten-free," "for a toddler." Pick needs your product genuinely serves.

This is the run most likely to tell you something you can act on. In Manic's sunscreen test, adding "under $25, for oily skin" took every model off the catalog's top product, and a listing that said "without clogging pores" won a third of one model's picks despite the lowest rating in its group. If your product serves the need but doesn't win, the reasons will usually show why: a competitor's listing says the words, and yours doesn't.

Step 5: Reverse the list

Run the baseline again with the list in reverse order, so the top product goes from first to last. Manic found the effect was strong on some searches and absent on most, and no model ever mentioned position in its reasons. You want to know which kind your category is. If position matters a lot, your catalog rank matters more, and the review and relevance signals that drive it deserve more attention.

Step 6: Tally picks and code the reasons

For each run, count picks per product per model. Then read every reason and tag what it cites.

Product	Model	Picks	Cites review count	Cites rating	Cites price	Cites a need phrase	Cites a claim	Exact phrase quoted
Your product	Model 1
Competitor A	Model 1
Your product	Model 2

The reasons are the most valuable output. They tell you which facts the model uses, which phrases it repeats, and whether it is repeating them correctly. In Manic's data, phrases like "clinically proven" and "11x faster" came straight from listings into the models' reasons. If a model is repeating something from your listing, make sure it is something you can stand behind; we argue why in today's think piece.

Step 7: Change one thing, then retest next month

Pick the single highest-value fix the test suggests:

• A wrong or odd catalog fact. Fix the source: your product data, your review app's sync, or your feed.
• A need your product serves but your listing doesn't state. Put it in the title, summary or first features, which is all the models in the test saw.
• A claim you would not want repeated. Soften it, substantiate it or remove it.

Shopify's own advice in its Muse post is the same in spirit: "Agents can only recommend what they understand," so clear titles, structured attributes, complete variants and reviews matter. Make the change, wait for the catalog to update, and rerun the same frozen-list method with a fresh list. Keep the requests and instruction identical so month-to-month results are comparable.

What this test can and can't tell you

It can show how models behave on your category's real catalog data, which of your facts and phrases they use, and whether a listing change moves picks. That is useful, cheap and repeatable.

It can't tell you what a specific app will do. ChatGPT, Gemini, Claude Duet and Meta's Muse each add their own instructions and, increasingly, the shopper's history. As we wrote in stop asking where you rank, there is no single position to own. Treat the test as a view of the layer underneath every app, not a ranking.

If you want this done continuously across many requests and assistants, XENA Intelligence builds that kind of tracking for consumer brands, with SearchPanel covering individual products. But the spreadsheet version above will answer the first question most brands have: when an agent chooses in our category, what does it read, and does it choose us?

A one-afternoon version

If you only have a few hours:

1. Choose your single most important request.
2. Save the top 20 catalog results.
3. Ask two different models to pick, 10 times each, fresh each time.
4. Add one real shopper need and repeat.
5. Read all 40 reasons and highlight every phrase that came from a listing.

That is enough to find your first fix.