SEO.to / 05
Robots.txt tester
Put in a domain. The tool loads its robots.txt, breaks it into rule groups, and lets you test any URL against any crawler. It uses the same longest-match rule that Google uses.
Fetched live. A bare domain works.
How the rules work
A crawler picks the one group that best matches its user-agent, then applies the longest matching rule to each URL. If an Allow and a Disallow are the same length, Allow wins. Groups do not stack: if there is a Googlebot group, Googlebot ignores the * group completely.
What each directive does
| Directive | Meaning |
|---|---|
User-agent | Starts a group. Multiple agents can share one group. |
Disallow | Path prefix to block. Empty value blocks nothing. |
Allow | Path prefix to permit; wins over an equal-length Disallow. |
* and $ | Wildcard and end-anchor, supported by all major engines. |
Sitemap | Absolute sitemap URL, outside any group. |
Crawl-delay | Ignored by Google; honoured by some other bots. |
robots.txt controls crawling, not indexing. A blocked URL can still end up in the index if other sites link to it. To keep a page out of the index, use noindex on the page itself.