The short answer
A good robots.txt for AI crawlers lets the AI search and user crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User) read your public pages, keeps only private paths out, and treats training crawlers (GPTBot, ClaudeBot) and control tokens (Google-Extended, Applebot-Extended) as a separate choice. Blocking a training crawler does not remove you from that company's AI search, so you can opt out of training and still be cited.
Below is a template you can copy, then what each user agent does according to the company that runs it, checked on 19 November 2026.
A robots.txt template for AI crawlers
Replace the private paths and the sitemap URL with your own:
# Everyone else, including Googlebot and Bingbot: public pages yes, private paths no
User-agent: *
Disallow: /admin
Disallow: /api/
Disallow: /checkout
# AI search and user-triggered fetchers: these decide whether assistants can cite you
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin
Disallow: /api/
Disallow: /checkout
# Model training: keep these lines to allow it, or change them to Disallow: / to opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /admin
Disallow: /api/
Disallow: /checkout
Sitemap: https://www.example.com/sitemap.xmlThree things make this template work:
- A crawler follows only the group that names it. If a group names GPTBot, GPTBot ignores the User-agent: * group, so every named group repeats the private paths.
- Several User-agent lines can share one group, which keeps the file short.
- The Sitemap line uses an absolute URL and sits outside any group.
Which AI crawlers to list, and what each one does
| User agent | Company | What it does | Follows robots.txt? |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT search answers | Yes: OpenAI says opted-out sites are not shown in ChatGPT search answers |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user or a custom GPT asks for it | Not always: OpenAI says robots.txt rules may not apply because a user started the visit |
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI models | Yes |
| Claude-SearchBot | Anthropic | Crawls to improve Claude's search results | Yes |
| Claude-User | Anthropic | Fetches a page when a Claude user asks a question | Yes |
| ClaudeBot | Anthropic | Collects content that may be used to train models | Yes |
| PerplexityBot | Perplexity | Finds and links pages in Perplexity answers; not used for training | Yes |
| Perplexity-User | Perplexity | Fetches a page when a Perplexity user asks a question | Generally no: Perplexity says it generally ignores robots.txt because a user asked |
| Google-Extended | Not a crawler: a token that controls whether Google uses content to train Gemini and to ground Gemini answers | Yes; Google says it does not affect inclusion or ranking in Google Search | |
| Applebot-Extended | Apple | Not a crawler: a token that controls whether Apple uses pages Applebot crawled to train its models | Yes; Apple says pages that disallow it can still appear in search results |
Sources: OpenAI's overview of its crawlers, Anthropic's help article on its bots, Perplexity's bot documentation, Google's list of common crawlers and Apple's Applebot page. Older Anthropic user agents (Claude-Web, anthropic-ai) are deprecated, but keeping them in an allow group does no harm.
Training versus search: the choice you actually make
For OpenAI, Anthropic, Google and Apple, the training switch and the search switch are separate:
- Disallowing GPTBot says your content should not train OpenAI models; OAI-SearchBot can still show you in ChatGPT search.
- Blocking ClaudeBot does not block Claude-SearchBot or Claude-User.
- Google-Extended does not change Google Search, and Applebot-Extended does not remove you from Apple's search results.
So the usual mistake is not opting out of training. It is a broad rule, such as Disallow: / under User-agent: *, or a named group without the right Allow, that also shuts out the search crawlers you need to be cited.
Mistakes we see most in AI crawler rules
- Disallow: / in a group that names OAI-SearchBot, Claude-SearchBot or PerplexityBot: that site cannot be found by those assistants' search.
- A named group that lists only Allow lines and forgets the private paths, which then become crawlable for that bot.
- Rules for user agents that no longer exist while the current ones are missing.
- robots.txt allowing a crawler while the CDN or firewall blocks it. Check both: see how to check if GPTBot and ClaudeBot can access your website.
Check your robots.txt
The free LLM Scan scan reads your robots.txt groups for the main AI crawlers, together with llms.txt, your sitemap and structured data, and tells you which crawler is blocked. To give assistants a map of your key pages once they get in, add an llms.txt file.