Where does robots.txt go?

At the root of the domain: https://example.com/robots.txt. Each subdomain needs its own file, and the file must be plain text served as HTTP 200.

Does blocking GPTBot stop ChatGPT citing me?

Blocking GPTBot only opts you out of model training. Citations today come from the search and user-agent crawlers like OAI-SearchBot and ChatGPT-User, which are separate bots.

Should I block admin paths?

Yes, mostly to keep crawl budget and error logs clean. The file is a request, not a security control, so never rely on it to hide anything sensitive.

SEO.to / 16

Robots.txt generator

Pick which crawlers may read the site, which paths to block, and paste your sitemap URL. Copy the finished file to the root of your domain.

robots.txt

Crawlers toggled off are disallowed for all paths. Test the result with the robots.txt tester.

What the file does

robots.txt tells well-behaved crawlers which paths they may request. It controls crawling, not indexing: a page blocked from crawling can still be indexed if other pages link to it.

The three decisions

Blocked paths: internal tools, drafts, faceted search parameters. Never block CSS or JavaScript, crawlers need them to render pages.
AI crawlers: toggling a bot off writes a disallow group for that user-agent. Search indexers like OAI-SearchBot are the ones whose blocking removes you from AI citations.
Sitemap: the Sitemap line is a pointer, not a rule, and it belongs outside any group.

Robots.txt generator

What the file does

The three decisions

Next