Technical ·
Are AI crawlers allowed to read your site?
Check robots.txt and your CDN bot settings, because a blocked crawler can't learn anything from your pages.
Most brands I look at have never opened their robots.txt. Some CDNs, Cloudflare included, now offer one-click blocking of AI bots, and it can be switched on without anyone in marketing knowing. If GPTBot, OAI-SearchBot, PerplexityBot or ClaudeBot get a 403, your own site drops out of the answer and the assistant leans on whatever Amazon or a reseller says about you.
Do this
- Open yoursite.com/robots.txt and look for Disallow lines aimed at AI user agents.
- Ask whoever runs your CDN or hosting to check the bot or AI crawler settings.
- Write down which bots you allow and why, so the decision is on purpose.