Why a page that opens in your browser can still be invisible to AI
Your browser is a person. AI crawlers are not, and sites treat them differently in three places. robots.txt can ask a crawler to stay out. A firewall or CDN bot setting can refuse it outright, often without anyone on the team knowing it was switched on. And the page can ask not to be indexed. Any one of the three keeps you out of the answers built from that crawler's reading.
This check tests all three for each crawler separately, because each is fixed in a different place: robots.txt in the file itself, the firewall in your CDN or host settings, and noindex in the page template.
The crawlersThree kinds of AI crawler, three different decisions
| Kind | Examples | Blocking it means |
|---|---|---|
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot | You drop out of the answers people get today. |
| Fetches for a user | ChatGPT-User, Claude-User, Perplexity-User | When someone pastes your link into a chat, the assistant cannot open it. |
| Model training | GPTBot, ClaudeBot, Google-Extended, CCBot | Your content stays out of future models. Current answers are not affected. |
A blanket "block AI bots" setting usually refuses all three. If you only want to keep your content out of training, block the training crawlers by name and leave search and user crawlers allowed.
FixesHow to fix what the check finds
If robots.txt blocks a crawler, the result names the exact line. Remove it, or add a group for the crawler you want to allow. A crawler that has its own group ignores the rules for all crawlers, so the fix can be as small as User-agent: OAI-SearchBot followed by Allow: /.
If the server blocks it, look for a bot or AI crawler setting in your CDN or host. In Cloudflare that is under Security, in the bot and AI crawler controls. Allow the search and user crawlers you want; verified bot lists are the safest way to do it.
If the page is marked noindex, remove the noindex from its robots meta tag or X-Robots-Tag header, unless it really should stay out of search.
If the page's HTML carries little text, it probably builds its content with JavaScript, and most AI crawlers see an empty shell. Server-side rendering or pre-rendering fixes it.
LimitsWhat this check cannot tell you
Requests come from our servers with each crawler's published user agent, not from the crawler's own network. A firewall rule keyed on IP addresses can treat the real crawler differently. And being readable is not the same as being cited: that depends on what the page says. To see which AI answers actually cite you, run the free AI visibility audit.
