Your robots.txt says AI is welcome. Your bot wall says no.

Most brand teams treat "AI readiness" as a content problem. Write clearer product copy. Add schema. Check what ChatGPT says about the SKU. Those jobs matter. They also assume an agent can load the page.

Blend Commerce spent a week in September testing that assumption. The CRO agency sent an AI shopping agent to 200 retailers, 100 in the UK and 100 in the US, across 20 sectors. It asked four questions: can the agent reach the site, find products, read the product data, and get a basket to the checkout. The average score was 65 out of 100.

The number that should stop a CPG meeting is smaller and sharper. Twenty-nine of those 200 stores refused the agent twice. That is 14.5%, or one in seven. The refused list includes Home Depot, Best Buy, Macy's, Tesco, Argos and Next. Four retailers (Saks Fifth Avenue, Bergdorf Goodman, Anthropologie and JD Williams) refused every request Blend made, home page and robots.txt included, and scored zero.

The door that says yes and the door that says no

Here is the part Blend says it did not expect. It could read robots.txt for 24 of the 29 refusing stores. Every one of those 24 allows the search crawlers from ChatGPT, Perplexity and Claude.

So the public crawler rules say AI is welcome. The software in front of the site still turned the shopping agent away with an error, a challenge page or an empty product page. Blend calls that bot protection: tools that judge visitors by how they behave, not only by what robots.txt prints.

Show the data

Size tier	Value
Large (16 of 80)	20%
Mid (12 of 80)	15%
Emerging (1 of 40)	2.5%

Source: Blend Commerce, AI Shopping Audit 2026 (audited Sep 8-15, 2026). A retailer may allow named AI agents while refusing automation it does not recognise.

Large retailers refused at 20%. Emerging ones at 2.5%. That pattern fits a boring organisational story: bigger companies have stronger security defaults, and nobody asked the revenue owner what should happen when an agent arrives ready to spend a customer's money.

PPC Land, covering the audit on November 5, quotes Blend co-founder Adam Pearce on the same gap: marketing teams may think AI agents are welcome because robots.txt says so, while the front-door software refuses them. His advice ahead of Black Friday was blunt. Security and marketing need the same room.

What the audit does not prove

Read the method before you panic.

Blend did not impersonate ChatGPT, Gemini, Copilot, Perplexity or Meta's Muse. Plain requests declared what they were. The browser was an unmodified Chromium. The agent was honest and unfamiliar. A caption under Blend's refusal chart says a retailer may allow named AI agents while refusing automation it does not recognise.

That matters. Amazon has publicly blocked Meta's Muse from shopping its site as a stated policy. Blend's refusals may be intentional policy, a vendor default, or a rule nobody has reviewed since it was set. The audit cannot tell those apart. It only shows that an unlisted automated visitor was refused twice from a residential connection in the store's own country.

Forter adds context from the other side of checkout. On its network, screen-scraping agents made 1% of agentic orders in late July 2026 and 77% in the week of October 19. Those agents drive the merchant's own pages. If your bot wall treats every unfamiliar automated browser as hostile, you are not only stopping scrapers. You may also be stopping the shopping path that is already growing fastest in Forter's sample.

Sector scores, and why CPG still cares

Sector	Average score out of 100
Baby and nursery	88
Furniture and beds	81
Jewellery and watches	79
Health and wellness	72
Beauty	69
Food and drink	58
Electronics	55
Department stores	51
Fashion	49
All 200 retailers	65

Source: Blend Commerce. Sector figures describe the ten retailers audited, not the whole sector.

Food and drink at 58 and beauty at 69 are not disasters. Fashion at 49 and department stores at 51 are worse. For a CPG brand the useful reading is not "our category won." It is "the retailers that carry us may be the ones refusing agents," and "our own DTC site may be doing the same thing without anyone in brand noticing."

Shopify stores in the audit averaged 91 against 59 for everyone else, helped by a default agent checkout endpoint. That is a platform story as much as a merchant story. It does not rescue a product page that never loads.

The silent sales leak

If an agent cannot load the product page, it cannot recommend what is on it, and it definitely cannot buy it. That sentence from Blend is the whole argument.

Brand teams spend money to be named in ChatGPT answers, to clean Merchant Center attributes, to publish returns copy. None of that helps when the browser session never reaches the SKU. PPC Land notes that Tesco, which appears on Blend's refused list, was still among ChatGPT's most-cited UK domains in other research. Visibility and reachability are separate problems.

We think of this as a silent sales leak because it rarely shows up in a dashboard. There is no "agent refused at WAF" row in most analytics. The shopper asked an assistant to buy. The assistant failed. The order went to whoever loaded.

For DTC brands the same audit is a mirror. You may not be Home Depot, but you may share a CDN rule written for credential stuffing that also trips an honest agent. Run the product-to-cart test from a phone on cellular data as well as from the office network. Some walls treat data-centre and residential traffic differently, and Blend's own method section notes connection-type limits between markets.

XENA Intelligence's SearchPanel is built to score how products show up in assistant answers. That work still assumes the page is fetchable when an agent later tries to act. A blocked storefront makes the visibility score academic.

What a CPG founder should do this week

You do not need a new platform. You need a conversation and a test.

1. Ask security which named AI shopping agents are on the bot-protection allow list, and who last changed that list. Tip 47 is the short version of that question.
2. Write down whether the rule is intentional policy (like Amazon's Muse block) or an unreviewed default.
3. Pick your best seller. Ask a current agent to open the product page and add it to a cart from a normal residential network. Record the first failure.
4. If you sell through retailers on Blend's refused list, treat that as a commercial fact: agent traffic for those doors may stop at the edge, whatever your listing quality.
5. After any CDN, WAF or bot-vendor change, rerun the same product-to-cart task. Configuration drifts.

Also check the quieter product-data gaps Blend found on pages it could read: only 18% of readable stores published a GTIN in product data, 22% shipping details, 20% a returns policy. We break that checklist down in the companion post. Access first, then facts.

The institutional fix

A few years ago, turning unknown automation away was an easy default. Fraud teams were hired to say no. Revenue teams were hired to grow traffic from people. Agents sit between those job descriptions. They spend like customers and arrive like bots.

Blend's survey of 59 ecommerce leaders (July 15 to August 18, 2026) fits the same picture: 63% said an AI agent could not complete their checkout, 61% had no 2026 budget for agent optimisation, and 57% said nobody owns AI readiness internally. The audit and the survey cover different businesses, but the organisational hole is the same.

The brands that get this right will not be the ones with the cleverest prompt copy. They will be the ones where security and revenue share a written rule about which agents get through the door, and where someone retests that rule before peak season.

Robots.txt is a public letter. Bot protection is the lock. If they disagree, the lock wins. Find out what your lock is doing.