Robots.txt let AI crawlers into 24 CPG brand sites. Their servers turned some of them away

A brand's robots.txt file is a sign on the door. It says which crawlers are welcome. It doesn't say whether the door actually opens when they knock. That second part is decided by the server, the CDN in front of it and whatever bot rules someone switched on two years ago.

Most AI visibility audits stop at the sign. We wanted to see the door, so we looked at what one of the open crawlers actually collected from a set of CPG brand sites.

Why Common Crawl

Common Crawl publishes a free archive of the web each month or so. Its October 2026 archive, released October 21, holds 2.16 billion pages crawled between October 5 and 18, from 41.4 million hosts and 34.3 million registered domains. Its crawler, CCBot, identifies itself and follows robots.txt, according to its documentation.

The archive matters for AEO because it's one of the ways web pages end up in training data. Graphite's research on parametric knowledge, published October 16, lists CCBot alongside GPTBot, ClaudeBot and Google-Extended as crawlers a brand should allow if it wants models to learn about it. We covered that study in our training-data playbook. This post is the check.

Common Crawl also does something the AI labs don't: it publishes an index of every URL it captured and the HTTP status code it got back. That lets anyone see, site by site, which requests worked.

What we checked

We picked 24 well-known direct-to-consumer and CPG brands across food, drinks, beauty, personal care and household goods. This is a convenience sample of brands we thought readers would recognize, not a random one, so the results describe these sites and not the industry.

For each site we did two things on October 27, 2026:

1. Fetched robots.txt and tested seven crawler names (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Google-Extended, CCBot and PerplexityBot) against the home page and a product path.
2. Queried Common Crawl's index for the October 2026 crawl (CC-MAIN-2026-43) and counted every capture for the domain and its subdomains, grouped by status code.

Everything below comes from our query of the Common Crawl index and the sites' robots.txt files on October 27, 2026. The robots.txt files we read are today's versions; the crawl ran earlier in October, so a site could have changed its file in between.

Robots.txt: everyone said yes

Every one of the 24 sites allowed all seven crawler names, on the home page and on product pages. One site, Poppi, returned no robots.txt at all, which crawlers treat as permission to crawl everything.

Only one file mentioned an AI crawler by name. Blueland's robots.txt has a User-agent: Google-Extended group with an empty Disallow: line, which explicitly allows it. None of the 24 files used a Content-Signal line, the newer robots.txt field for telling automated systems how a site's content may be used.

So on paper, these brands are as open to AI crawlers as a site can be. If your own audit ended here, you'd mark the crawler box green and move on.

The server responses: not everyone got in

Across the 24 sites, Common Crawl's October index holds 9,277 captures. Most were ordinary: 6,555 returned 200 (a normal page) and most of the rest were redirects or missing pages. But 259 returned 429, "Too Many Requests", and 33 returned 403, "Forbidden". Together that's 292 captures, or 3.1% of the total (our calculation).

A 429 is the server telling the crawler to slow down. It's defined in RFC 6585, and it's a normal tool for protecting a site from heavy traffic. But for the purposes of this archive, a page that got a 429 wasn't saved on that attempt. A 403 is a flat refusal.

The 3.1% average hides how uneven it was.

Show the data

Site (captures)	429 or 403 responses, share of captures
olly.com (13)	61.5%
eatfishwife.com (28)	46.4%
liquiddeath.com (98)	45.9%
athleticbrewing.com (90)	35.6%
mudwtr.com (35)	34.3%
magicspoon.com (50)	32%
kindsnacks.com (39)	30.8%
graza.co (54)	27.8%
drinkolipop.com (11)	27.3%
nativecos.com (38)	18.4%
glossier.com (398)	16.6%
eatbanza.com (108)	12%
drinkpoppi.com (43)	9.3%
kettleandfire.com (110)	5.5%
methodhome.com (48)	4.2%
huel.com (892)	2.9%
sietefoods.com (274)	1.8%
chobani.com (315)	1.3%
blueland.com (189)	0.5%
seventhgeneration.com (200)	0.5%
oatly.com (1870)	0.1%
drsquatch.com (254)	0%
simplemills.com (353)	0%
theordinary.com (3767)	0%

Source: Our query of the Common Crawl index (CC-MAIN-2026-43) on October 27, 2026. Our calculation: captures with status 429 or 403 divided by all captures for the domain, including subdomains. Sites with few captures (OLLY 13, Olipop 11) swing a lot on a handful of responses.

At 9 of the 24 sites, a quarter or more of the crawler's captures were 429s or 403s. At 5 sites it was between 5% and 25%. At 10 it was under 5%, and three sites returned none at all.

What got turned away

Looking at the refused URLs, they weren't obscure. At Athletic Brewing, 429s included product pages such as a black IPA and collection pages. At Liquid Death, they included collection pages and localized product pages. At OLLY, the 403s included the robots.txt file itself, plus help center articles on a subdomain. At KIND, 12 of 39 captures were 403s.

We can't see why any individual server answered the way it did. A 429 might come from the store platform's default rate limit, a CDN rule or a temporary traffic spike. A 403 could be a firewall rule aimed at bots in general rather than at CCBot in particular. None of this means a brand chose to block AI training. It means the request didn't get through, which for the archive comes to the same thing.

How much of each site was saved

The other way to read the index is to count what did get saved: unique HTML pages that returned a normal 200 response.

Table: CPG brand sites in Common Crawl's October 2026 crawl, sorted by HTML pages saved (domain and subdomains)

Site	Captures	Unique HTML pages saved (200)	429 responses	403 responses
theordinary.com	3,767	2,806	0	0
oatly.com	1,870	1,305	0	1
huel.com	892	603	26	0
chobani.com	315	212	0	4
drsquatch.com	254	187	0	0
sietefoods.com	274	173	2	3
simplemills.com	353	150	0	0
blueland.com	189	147	1	0
seventhgeneration.com	200	146	0	1
glossier.com	398	104	66	0
kettleandfire.com	110	79	6	0
eatbanza.com	108	61	13	0
athleticbrewing.com	90	37	32	0
graza.co	54	19	15	0
liquiddeath.com	98	19	45	0
mudwtr.com	35	16	12	0
magicspoon.com	50	12	16	0
nativecos.com	38	11	7	0
eatfishwife.com	28	7	13	0
kindsnacks.com	39	4	0	12
methodhome.com	48	3	0	2
drinkolipop.com	11	1	3	0
olly.com	13	1	2	6
drinkpoppi.com	43	0	0	4

The median site had 49 unique HTML pages saved (our calculation). For scale, the whole October archive averages about 63 pages per registered domain, which we got by dividing 2.16 billion pages by 34.3 million domains. Common Crawl samples the web rather than copying every page, so a low count isn't by itself a sign of trouble. Big sites with many country versions, like The Ordinary and Oatly, naturally show up more.

What stands out is the bottom of the table. Poppi had zero normal HTML pages saved; most of its 43 captures were missing pages (404s) and redirects, plus four 403s. Olipop and OLLY had one each. Method had three. For a model that learns partly from this archive, those brands' own websites barely exist in the October snapshot. What it knows about them would come from retailers, press and social posts instead.

The counts include subdomains, which changes the picture for a few sites. About half of Chobani's captures, for example, were on its careers subdomain. Product pages are a smaller share than the totals suggest.

What this means for a CPG brand

Our view: robots.txt audits are necessary and not sufficient. Graphite makes the same point in its research, advising brands to confirm that CDN or firewall bot rules aren't blocking crawlers, because robots.txt alone doesn't settle it. Our data is a small, concrete example of the gap.

There's also a timing reason to look now. Cloudflare's documentation says that since September 15, 2026, new domains on its network block crawlers it classifies as Training or Agent by default on pages that display ads. Platform defaults like that change what crawlers can reach without anyone on the brand team touching a setting.

How to check your own site

This takes about 20 minutes and no special tools.

1. Read your robots.txt as a crawler would

Open yourdomain.com/robots.txt and look for any group that names GPTBot, ClaudeBot, CCBot, Google-Extended, OAI-SearchBot or PerplexityBot, or a User-agent: * group with broad Disallow rules. Our tip on crawler access lists what each name does.

2. Look yourself up in Common Crawl

Go to index.commoncrawl.org, pick the most recent crawl and search for your domain with a wildcard. The results list every URL captured and its status code. Count the 200s, 429s and 403s. If your product pages appear only with 429s, the crawler knocked and wasn't let in.

3. Check your own logs

Your server or CDN logs show the same thing from your side, for every crawler, not just CCBot. Filter by user agent and look for 403 and 429 responses. Our tip on checking crawler visits walks through it.

4. Ask whoever runs the CDN

If you see refusals, find out which rule produced them. It might be a rate limit that's reasonable for scrapers but too tight for a crawler that respects robots.txt. Many CDNs, including Cloudflare, let you treat verified crawlers differently from unknown bots.

5. Make sure what does get saved is useful

A crawler that gets a 200 but finds an empty page because product details load in JavaScript hasn't learned much either. Our JavaScript tip has a quick test, and our grocery-site readability check shows what that looks like on real pages.

What this doesn't tell you

Common Crawl is one archive among many inputs to model training, and the labs don't publish exactly which snapshots they use. OpenAI, Anthropic and Google run their own crawlers, which may get different responses from the same servers. Some of the 429s here may have been retried successfully in another crawl. And 24 hand-picked sites are a small sample.

What the check does tell you is that permission and access are separate questions, and that the second one is easy to look up. Crawler access is also only the input. The output is what assistants actually say about your products, which is a different measurement: you ask the assistants and read the answers. Some brands do that by hand; XENA Intelligence does it as a service, asking assistants such as ChatGPT, Perplexity, Gemini and Amazon's assistant shopper-style questions and scoring products on how they show up, with SearchPanel showing the actual answers for each SKU. Either way, it's worth knowing whether the crawler got in before you try to explain the answers.