SEO.to / 12
AI crawler checker
Put in a domain. The tool reads its robots.txt and llms.txt and tells you which AI crawlers are allowed in, and whether that means the site can still be cited by assistants like ChatGPT.
Fetched live. A bare domain works.
The three kinds of AI bot
| Kind | Examples | Blocking means |
|---|---|---|
| Training crawlers | GPTBot, ClaudeBot, Google-Extended, CCBot, meta-externalagent | Your content stays out of future model training. No effect on being cited today. |
| Search indexers | OAI-SearchBot, Claude-SearchBot, PerplexityBot | You disappear from AI search products and their citations. |
| User fetchers | ChatGPT-User, Claude-User, Perplexity-User | Assistants cannot open your pages even when a human asks. |
A lot of sites block all three types with one rule, then wonder why assistants stopped citing them. Blocking the training crawlers while leaving the other two allowed is a reasonable setup, but most sites end up there by accident rather than on purpose.
llms.txt
This is a markdown file at /llms.txt that describes your site and lists its main pages for AI assistants. It is a new convention and not many sites have one yet.