PickedByRobots

No. 1 · Updated

Home / Tips / No. 31

Measurement ·

How do you check whether AI crawlers visit your site?

Filter server or CDN logs by AI crawler user agents and look for blocked (403) responses.

Logs tell you who actually came, which pages they hit and whether they got turned away. Cloudflare and most hosts can show bot traffic by user agent. No logs? Ask your developer. It's a 20-minute job.

Do this

  1. Pull a week of logs or a bot analytics report.
  2. Filter for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Bingbot.
  3. Look for 403s and pages they never reach.

More on measurement