The short answer
To check whether AI crawlers like GPTBot and ClaudeBot can access your website, read the rules for each crawler in your robots.txt, make sure your CDN or firewall does not block their user agents, and look for their visits in your server logs. The free LLM Scan scanner reads the robots.txt rules for 6 AI crawlers by name in about a minute, without an account.
robots.txt says what crawlers are allowed to do, the firewall decides what actually gets through, and the logs show what really happened.
Which AI crawlers to check
Each AI company uses its own user agents, usually one to collect training data, one to index for search and one to fetch a page when a user asks. robots.txt rules match the user agent name, so a rule for GPTBot does not cover OAI-SearchBot or ClaudeBot. Check each one by name.
| User agent | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train models |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks for it |
| ClaudeBot | Anthropic | Collects content that may be used to train models |
| Claude-SearchBot and Claude-User | Anthropic | Index pages for search and fetch a page when a Claude user asks for it |
| Claude-Web | Anthropic | Older user agent; keep it in your rules |
| PerplexityBot and Perplexity-User | Perplexity | Index pages for Perplexity answers and fetch a page when a user asks |
| Google-Extended | Not a separate crawler: a robots.txt token that controls whether Google uses content for Gemini training. Google Search and AI Overviews follow Googlebot rules |
robots.txt for AI crawlers: what a good file looks like
If you want AI assistants to describe and cite you, a good robots.txt for AI crawlers allows the search and user crawlers on your public pages and keeps only private paths out. You can still opt out of model training separately, by disallowing GPTBot, ClaudeBot or Google-Extended, without leaving AI search.
- Give a crawler its own User-agent group only when you want to treat it differently from the * group, because a named group replaces the * rules for that crawler.
- Disallow private paths such as /admin, /api/ and /checkout in every group you write, not only under *.
- Never Disallow: / for OAI-SearchBot, ChatGPT-User, Claude-SearchBot or PerplexityBot if you want to appear in their answers.
- Add an absolute Sitemap: line so crawlers find your canonical pages.
Step 1: Read your robots.txt per crawler
Open yoursite.com/robots.txt and find the group that applies to each crawler. A crawler follows the group that names it. Only when no group names it does it fall back to the User-agent: * group. This is where most surprises come from: a site that allows everything under User-agent: * can still block GPTBot with its own group further down.
What to look for:
- Disallow: / under a crawler's group blocks it from the whole site.
- Disallow on public sections such as /blog, /docs, /pricing or /solutions hides exactly the pages AI assistants need to describe you.
- Disallow on private paths such as /admin, /api/ or /checkout is normal and safe to keep.
- Lines without a colon, rules placed before any User-agent line, or relative Sitemap URLs can make crawlers ignore parts of the file.
The LLM Scan robots.txt check does this for GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot and Google-Extended. It fails the scan when any of them is blocked from the whole site. It warns when one of them is blocked from public sections or when the file has formatting errors. It names the crawler and the rule to change.
Step 2: Check your CDN and firewall
robots.txt is a request, not a gate. Your CDN, WAF or hosting provider can still refuse an AI crawler that robots.txt allows. Many CDNs and security tools have a setting that blocks AI bots. Look in your CDN's bot or security settings for rules that match AI crawlers, and check your firewall logs for blocked requests from those user agents.
A quick test is to request your homepage with a crawler's user agent:
- curl -I -A "Mozilla/5.0 (compatible; GPTBot/1.2)" https://yoursite.com/
A 403 or a challenge page means something in front of your site blocks that user agent. A 200 is a good sign but not proof. Some providers check the crawler's IP address as well as its user agent, so a fake request from your laptop can be treated differently from the real crawler.
Ways to check, compared
| Tool | What it checks | What it misses | Cost |
|---|---|---|---|
| LLM Scan free scan | robots.txt groups for GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot and Google-Extended, plus llms.txt, sitemap, structured data and raw HTML | CDN and firewall blocks, OAI-SearchBot, real crawler visits | Free, no account |
| SEOptimer AI Crawler Blocked Checker | Whether your robots.txt blocks the major AI crawlers | CDN and firewall blocks, real crawler visits; check their page for the bots it covers | Free |
| Other single-purpose AI crawler checkers | robots.txt permissions for a list of AI bots; some also request your page with a crawler user agent | Their requests come from their own servers, not from the AI companies' networks | Usually free |
| curl with a crawler user agent | What your server or CDN returns to that user agent | Real crawler IP checks, so a pass is not proof | Free |
| Server logs or LLM Scan crawler tracking | Which AI crawlers really visit, which pages and which status codes | Whether AI answers mention you | Free with your logs; tracking needs an LLM Scan workspace |
Step 3: Look for real visits in your server logs
The only proof that a crawler can reach you is a real visit. Search your access logs for the user agent names above and check the status codes they got. Repeated 403, 429 or 5xx responses to AI crawlers mean they are trying and failing.
Analytics scripts will not show you this. AI crawlers do not run JavaScript, so a browser-based analytics tag never sees them. The visit has to be recorded on the server. In an LLM Scan workspace, each monitored domain has a "Track AI crawlers" setup with a site token and snippets for Next.js, Cloudflare Workers and plain HTTP. Your server reports each crawler visit, and the dashboard shows which crawlers came and when.
After access: can AI actually read the page?
Access is only the first layer. Once a crawler gets in, it needs content it can read without running JavaScript, a sitemap that lists your canonical pages, and, if you publish one, an llms.txt that points to the pages that matter. The same free scan checks your sitemap, llms.txt, structured data and semantic HTML. See Fixing llms.txt and Sitemap Gaps for the details.
Being crawlable does not mean being recommended. To find out whether ChatGPT or Perplexity actually name you when buyers ask, you need to track the answers themselves. See how to measure AI brand mentions and citations.