← Back to blog
crawler-readinessrobots.txtguide

robots.txt for AI Crawlers: Copy-Paste Template

A copy-paste robots.txt for AI crawlers, and what OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and Google-Extended do, per their makers.

LLM Scan Team Published November 19, 2026

The short answer

A good robots.txt for AI crawlers lets the AI search and user crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User) read your public pages, keeps only private paths out, and treats training crawlers (GPTBot, ClaudeBot) and control tokens (Google-Extended, Applebot-Extended) as a separate choice. Blocking a training crawler does not remove you from that company's AI search, so you can opt out of training and still be cited.

Below is a template you can copy, then what each user agent does according to the company that runs it, checked on 19 November 2026.

A robots.txt template for AI crawlers

Replace the private paths and the sitemap URL with your own:

# Everyone else, including Googlebot and Bingbot: public pages yes, private paths no
User-agent: *
Disallow: /admin
Disallow: /api/
Disallow: /checkout

# AI search and user-triggered fetchers: these decide whether assistants can cite you
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin
Disallow: /api/
Disallow: /checkout

# Model training: keep these lines to allow it, or change them to Disallow: / to opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /admin
Disallow: /api/
Disallow: /checkout

Sitemap: https://www.example.com/sitemap.xml

Three things make this template work:

  • A crawler follows only the group that names it. If a group names GPTBot, GPTBot ignores the User-agent: * group, so every named group repeats the private paths.
  • Several User-agent lines can share one group, which keeps the file short.
  • The Sitemap line uses an absolute URL and sits outside any group.

Which AI crawlers to list, and what each one does

User agentCompanyWhat it doesFollows robots.txt?
OAI-SearchBotOpenAIFinds pages to show in ChatGPT search answersYes: OpenAI says opted-out sites are not shown in ChatGPT search answers
ChatGPT-UserOpenAIVisits a page when a ChatGPT user or a custom GPT asks for itNot always: OpenAI says robots.txt rules may not apply because a user started the visit
GPTBotOpenAICrawls content that may be used to train OpenAI modelsYes
Claude-SearchBotAnthropicCrawls to improve Claude's search resultsYes
Claude-UserAnthropicFetches a page when a Claude user asks a questionYes
ClaudeBotAnthropicCollects content that may be used to train modelsYes
PerplexityBotPerplexityFinds and links pages in Perplexity answers; not used for trainingYes
Perplexity-UserPerplexityFetches a page when a Perplexity user asks a questionGenerally no: Perplexity says it generally ignores robots.txt because a user asked
Google-ExtendedGoogleNot a crawler: a token that controls whether Google uses content to train Gemini and to ground Gemini answersYes; Google says it does not affect inclusion or ranking in Google Search
Applebot-ExtendedAppleNot a crawler: a token that controls whether Apple uses pages Applebot crawled to train its modelsYes; Apple says pages that disallow it can still appear in search results

Sources: OpenAI's overview of its crawlers, Anthropic's help article on its bots, Perplexity's bot documentation, Google's list of common crawlers and Apple's Applebot page. Older Anthropic user agents (Claude-Web, anthropic-ai) are deprecated, but keeping them in an allow group does no harm.

Training versus search: the choice you actually make

For OpenAI, Anthropic, Google and Apple, the training switch and the search switch are separate:

  • Disallowing GPTBot says your content should not train OpenAI models; OAI-SearchBot can still show you in ChatGPT search.
  • Blocking ClaudeBot does not block Claude-SearchBot or Claude-User.
  • Google-Extended does not change Google Search, and Applebot-Extended does not remove you from Apple's search results.

So the usual mistake is not opting out of training. It is a broad rule, such as Disallow: / under User-agent: *, or a named group without the right Allow, that also shuts out the search crawlers you need to be cited.

Mistakes we see most in AI crawler rules

  • Disallow: / in a group that names OAI-SearchBot, Claude-SearchBot or PerplexityBot: that site cannot be found by those assistants' search.
  • A named group that lists only Allow lines and forgets the private paths, which then become crawlable for that bot.
  • Rules for user agents that no longer exist while the current ones are missing.
  • robots.txt allowing a crawler while the CDN or firewall blocks it. Check both: see how to check if GPTBot and ClaudeBot can access your website.

Check your robots.txt

The free LLM Scan scan reads your robots.txt groups for the main AI crawlers, together with llms.txt, your sitemap and structured data, and tells you which crawler is blocked. To give assistants a map of your key pages once they get in, add an llms.txt file.

Questions and answers

What is the best robots.txt for AI crawlers?
One that lets OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User read your public pages, keeps private paths out of every group, and makes a deliberate choice about training crawlers such as GPTBot and ClaudeBot.
Can I block AI training but still appear in ChatGPT and Claude answers?
Yes. Disallow GPTBot and ClaudeBot and keep OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User allowed. OpenAI and Anthropic document these as separate user agents.
Does blocking Google-Extended remove me from Google AI Overviews?
Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls Gemini training and grounding. AI Overviews are part of Search, which Googlebot crawls.
Do all AI crawlers follow robots.txt?
No. OpenAI says robots.txt may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores it, because a person asked for the page. Anthropic says its three bots honor robots.txt.

See how AI assistants answer about your brand, every week

LLM Scan runs the questions your customers ask through AI assistants such as ChatGPT and Perplexity, and checks your Google rank and AI Overview citations for the keywords you track. Every week you get one report: mention rate and position by assistant, prompts won and lost, share of voice against competitors, the sources AI cites, and the 3 things to fix next.

// Start free

Find out what AI tells your customers about you.

Start with a free scan of your site. Paid plans add the weekly AI visibility report.