The short answer
The tools for llms.txt and AI crawler readiness fall into five groups: an llms.txt checker, a robots.txt reader that understands AI user agents, your CDN or firewall bot settings, a structured data validator, and server-side crawler tracking that shows which AI bots actually visit. LLM Scan's free llms.txt checker and AI readiness scanner cover the first, second and fourth in one run, and drafts an llms.txt, robots.txt additions and JSON-LD when something is missing.
The tools at a glance
| Tool | What it checks | What it misses | Cost |
|---|---|---|---|
| LLM Scan llms.txt checker and scanner | llms.txt quality, robots.txt for 6 AI crawlers, sitemap, JSON-LD, raw HTML, Markdown support | Whole-site crawl, JavaScript rendering, firewall blocks, OAI-SearchBot | Free, no account |
| curl with an AI user agent | What your server or CDN returns to a given user agent | Real crawler IPs, so a pass is not proof | Free |
| Google Search Console robots.txt report | Whether Google can fetch and parse robots.txt | Rules for non-Google AI crawlers | Free |
| CDN and firewall bot settings | Whether AI bots are blocked or challenged before robots.txt | What the bots do with your content | Included with your CDN; check your provider |
| Rich Results Test and Schema.org validator | JSON-LD validity and eligible rich results | Whether the markup matches your positioning | Free |
| Server logs or LLM Scan crawler tracking | Which AI crawlers really visit, which pages, which status codes | Whether AI answers mention you | Free with your logs; tracking needs an LLM Scan workspace |
| AI visibility platforms with crawl audits (for example Peec AI) | Crawlability across many AI bots, next to answer tracking | Check their docs | Check their pricing page |
1. An llms.txt checker
llms.txt is a short Markdown file at /llms.txt that points language models to your important pages. It is an emerging convention, so check that yours is useful rather than just present. LLM Scan's checker passes a file that is served as text, has more than 200 characters, at least one heading and at least one absolute URL, and drafts one for you when it fails. Review the draft before publishing: it should link your product, pricing, docs and policy pages, not every URL.
2. A robots.txt reader for AI user agents
Generic robots.txt testers often only evaluate Googlebot. For AI readiness you need the rules for each AI crawler by name, because a crawler follows the group that names it and ignores the * group. LLM Scan reads the groups for GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot and Google-Extended, fails the scan when one is blocked from the whole site, and warns about blocks on public sections and formatting errors. Check OAI-SearchBot by hand if ChatGPT search matters to you.
3. Your CDN and firewall
robots.txt is a request; your CDN decides what gets through. Many CDNs and security tools can block or challenge AI bots, sometimes by default. Look in your provider's bot settings, then test with curl:
- curl -I -A "Mozilla/5.0 (compatible; GPTBot/1.2)" https://yoursite.com/
A 403 or a challenge page means something in front of your site blocks that user agent. Our guide on checking GPTBot and ClaudeBot access covers the details.
4. A structured data validator
Google's Rich Results Test and the Schema.org validator tell you whether your JSON-LD parses. LLM Scan passes structured data when a valid JSON-LD block includes Organization or WebSite, and drafts the markup when it is missing.
5. Server-side crawler tracking
The only proof that AI crawlers can reach you is a real visit. Analytics scripts cannot see crawlers because crawlers do not run JavaScript. Read your access logs, or report visits from your server: in an LLM Scan workspace each monitored domain gets a site token and snippets for Next.js, Cloudflare Workers and plain HTTP. It recognizes GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Meta-ExternalAgent, cohere-ai and Bingbot, and shows which pages they requested.
Readiness is not visibility
All of these tools answer one question: can AI read your site? None tells you whether ChatGPT or Perplexity recommend you. For that you need to track the answers to your buyers' questions; see the best tools to track brand visibility in AI search. For the full step-by-step check, read how to check whether your website is ready for LLMs.