How to check whether AI crawlers can read your site
The short version. Fetch your robots.txt and read it against the actual AI crawler user agents, then fetch a real page using one of those agents from outside your network. If either step fails, nothing else you do for AI visibility can work.
What you need
- Your domain.
- A terminal, or any tool that can send a custom user agent. A browser alone is not enough, because a browser cannot pretend to be a crawler convincingly.
01Read your robots.txt against the real agent list
Open yourdomain.com/robots.txt and read every User-agent block, not just the first one. You are looking for any Disallow that applies to the AI crawlers rather than only to classic search bots.
The agents worth checking by name are GPTBot, OAI-SearchBot and ChatGPT-User for ChatGPT; PerplexityBot and Perplexity-User for Perplexity; ClaudeBot and Claude-User for Claude; Google-Extended for Gemini; and Googlebot, which is also the crawler behind AI Overviews and AI Mode. Applebot-Extended, CCBot, meta-externalagent, Bytespider and Amazonbot feed broader corpora.
Note the distinction as you read: training and corpus crawlers affect what a model absorbed over time, while index and live-fetch agents can remove you from answers immediately. Blocking the second group is the expensive mistake.
02Watch for the prefix-matching trap
Robots.txt user-agent tokens are matched case-insensitively as a prefix of the declared agent. This causes two silent failures in opposite directions.
A rule naming a short token can catch more agents than intended. And a rule with a typo, or one naming an agent that does not exist under that spelling, matches nothing at all while looking like a working policy. Neither failure is visible without parsing the file against the real agent list.
03Fetch a real page as a bot, from outside your network
robots.txt is a request, not an enforcement layer, and it tells you nothing about what your infrastructure does. Bot protection, WAF rules, CDN challenges and rate limits all sit above it and can reject a crawler that robots.txt welcomes.
Request one of your important pages with a crawler user agent, from a network that is not your office and not signed in. A 200 with your actual content is a pass. A challenge page, a 403, a login wall, or a near-empty shell is a failure, regardless of what robots.txt says.
04Compare what came back against what a browser renders
A successful fetch is not the same as extractable content. Read what the crawler received and look for the claims that matter: what you do, who you serve, what you cost, why you are different.
If those exist in the browser but not in the fetched response, the content is arriving via client-side JavaScript, or living inside images with no text equivalent. Both are extraction failures, and both are invisible from your own screen.
05Fix in order of cost, and re-check
Work in this order: remove or narrow blocking robots.txt rules, then allowlist the crawler agents in whatever bot protection sits in front of the site, then server-render the pages carrying your important claims.
Then run the same fetch again. This is the one part of AI visibility work with an immediate, verifiable result, which is why it is worth doing before any slower effort to earn mentions.
What this method does not do
- A pass here does not mean you will be cited. It means you are eligible to be. Retrievability is a gate, not a lever.
- Crawler behavior and user agents change. A check is true on the day you run it, which is an argument for re-running it after any infrastructure change.
Every step above works without buying anything. If you would rather have it run repeatedly with the sample sizes attached, our free report does the measurement on your own domain.