How can I check if GPTBot and ClaudeBot can access my website?

Read the robots.txt group that applies to each crawler, make sure your CDN or firewall does not block their user agents, and look for their visits in your server logs. The free LLM Scan scan checks robots.txt for GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot and Google-Extended.

Does allowing User-agent: * also allow GPTBot?

Only if robots.txt has no group that names GPTBot. A crawler follows the group with its own name and ignores the * group, so a specific GPTBot group overrides a permissive global rule.

Can my site block AI crawlers even if robots.txt allows them?

Yes. A CDN, firewall or hosting provider can reject AI crawler user agents before robots.txt matters. Check your bot or security settings and look for 403 responses to AI crawlers in your logs.

Why don't AI crawlers show up in my analytics?

AI crawlers do not run JavaScript, so browser-based analytics never record them. Use server logs or report each visit from your server, as LLM Scan's crawler tracking does.

Is it safe to block AI crawlers from some paths?

Yes. Excluding private paths such as /admin, /api/ or /checkout is normal. Avoid blocking public sections such as /blog, /docs, /pricing or /solutions, because those are the pages assistants use to describe you.

How to Check if AI Crawlers Like GPTBot and ClaudeBot Can Access Your Website

The short answer

To check whether AI crawlers like GPTBot and ClaudeBot can access your website, read the rules for each crawler in your robots.txt, make sure your CDN or firewall does not block their user agents, and look for their visits in your server logs. robots.txt says what crawlers are allowed to do, the firewall decides what actually gets through, and the logs show what really happened. The free LLM Scan scanner runs the first check for you.

Which AI crawlers to check

Each AI company uses its own user agent, and some use more than one:

GPTBot and ChatGPT-User (OpenAI)
ClaudeBot and Claude-Web (Anthropic)
PerplexityBot (Perplexity)
Google-Extended (Google's token for AI use of your content)

robots.txt rules match the user agent name, so a rule for GPTBot does not cover ClaudeBot. Check each one by name.

Step 1: Read your robots.txt per crawler

Open yoursite.com/robots.txt and find the group that applies to each crawler. A crawler follows the group that names it. Only when no group names it does it fall back to the User-agent: * group. This is where most surprises come from: a site that allows everything under User-agent: * can still block GPTBot with its own group further down.

What to look for:

Disallow: / under a crawler's group blocks it from the whole site.
Disallow on public sections such as /blog, /docs, /pricing or /solutions hides exactly the pages AI assistants need to describe you.
Disallow on private paths such as /admin, /api/ or /checkout is normal and safe to keep.
Lines without a colon, rules placed before any User-agent line, or relative Sitemap URLs can make crawlers ignore parts of the file.

The LLM Scan robots.txt check does this for GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot and Google-Extended. It fails the scan when any of them is blocked from the whole site. It warns when one of them is blocked from public sections or when the file has formatting errors. It names the crawler and the rule to change.

Step 2: Check your CDN and firewall

robots.txt is a request, not a gate. Your CDN, WAF or hosting provider can still refuse an AI crawler that robots.txt allows. Many CDNs and security tools have a setting that blocks AI bots. Look in your CDN's bot or security settings for rules that match AI crawlers, and check your firewall logs for blocked requests from those user agents.

A quick test is to request your homepage with a crawler's user agent:

curl -I -A "Mozilla/5.0 (compatible; GPTBot/1.2)" https://yoursite.com/

A 403 or a challenge page means something in front of your site blocks that user agent. A 200 is a good sign but not proof. Some providers check the crawler's IP address as well as its user agent, so a fake request from your laptop can be treated differently from the real crawler.

Step 3: Look for real visits in your server logs

The only proof that a crawler can reach you is a real visit. Search your access logs for the user agent names above and check the status codes they got. Repeated 403, 429 or 5xx responses to AI crawlers mean they are trying and failing.

Analytics scripts will not show you this. AI crawlers do not run JavaScript, so a browser-based analytics tag never sees them. The visit has to be recorded on the server. In an LLM Scan workspace, each monitored domain has a "Track AI crawlers" setup with a site token and snippets for Next.js, Cloudflare Workers and plain HTTP. Your server reports each crawler visit, and the dashboard shows which crawlers came and when.

After access: can AI actually read the page?

Access is only the first layer. Once a crawler gets in, it needs content it can read without running JavaScript, a sitemap that lists your canonical pages, and, if you publish one, an llms.txt that points to the pages that matter. The same free scan checks your sitemap, llms.txt, structured data and semantic HTML. See Fixing llms.txt and Sitemap Gaps for the details.

Being crawlable does not mean being recommended. To find out whether ChatGPT or Perplexity actually name you when buyers ask, you need to track the answers themselves. See how to measure AI brand mentions and citations.