Ultimate Guide to Robots.txt (2026)

robots.txt is a plain text file at the root of your domain that tells well-behaved crawlers which paths they may fetch and which they may not. It is also the single most misunderstood file on the average website, because most people treat it as a security wall when it is actually a polite suggestion, and they use it to block indexing when it only ever controlled crawling. This guide is the complete reference: every directive Google actually supports, the longest-match algorithm that decides who wins when rules collide, the full 2026 AI crawler matrix, and the exact mistakes that silently keep pages out of Google or hand your content to model training without you noticing. Every rule here pairs with a live check you can run in the robots.txt tester and every claim links to a primary source, so you can verify each point instead of trusting a blog that copied another blog.

What Robots.txt Controls in 2026 (and What It Cannot Do)

robots.txt controls exactly one thing in practice: which well-behaved crawlers are allowed to fetch which URL paths on your origin. A crawler that honors the Robots Exclusion Protocol requests the file before it crawls, parses the rules, and skips the paths you have disallowed. That is the entire contract. The file does not block people, does not stop browsers, does not authenticate anything, and does not remove a page from search results by itself.

The part most site owners get wrong is the boundary between crawling and indexing. Crawling is the act of fetching a page. Indexing is the act of storing it and being willing to show it in results. robots.txt governs the first and only the first. Google's robots.txt introduction (2025) states this in plain terms, and it is the reason a page you have blocked in robots.txt can still show up in Google when other sites link to it. We return to that trap in detail in the Disallow versus noindex section, because it is the most expensive mistake in this whole guide.

What the file can do is narrower than most people hope. It can steer Googlebot away from your internal search pages, your staging environment, your shopping cart, and your PDF archives. It can declare your sitemap location. In 2026 it can also make a per-bot decision about AI crawlers, letting you opt out of model training while staying in search results and AI citations. What it cannot do is enforce any of that against a crawler that ignores the file entirely.

You should still write the file carefully. A broken or over-broad robots.txt is one of the few things that can quietly remove an entire section of your site from search, and it takes weeks to notice because the damage is absence, not an error message. The sections that follow give you the exact grammar and the exact testing workflow so you never ship a rule you have not verified.

RFC 9309: The Robots Exclusion Protocol Standard, Explained

The Robots Exclusion Protocol began as a proposal by Martijn Koster in June 1994, long before Google existed, as a way for site owners to tell early web crawlers where they were unwelcome. For nearly three decades it stayed a de facto convention: every crawler implemented roughly the same grammar, but the edges differed. That changed in September 2022, when the protocol was standardized as RFC 9309 (2022).

Standardization matters for one practical reason: it fixed the parsing rules. RFC 9309 defines the exact grammar of a robots.txt file, how groups of user-agent lines are paired with allow and disallow rules, how a crawler must match a URL against those rules, and how the longest match wins. It also defines caching behavior, so a crawler can hold the file for a time instead of re-fetching it on every request. Two things the RFC treats as extensions rather than core rules are crawl-delay and sitemap, which the standard documents but does not require every crawler to honor. RFC 9309 (2022) is the reference to quote when a developer tells you your file is "wrong" without having read the spec.

Google then aligned its own guidance with the RFC. In February and March 2025 Google ran a Robots Refresher series (2025) to re-explain the fundamentals, and it stated that its interpretation of the file is based on RFC 9309. The practical upshot is that a file written to the RFC parses the same way in Google, Bing, and every crawler that claims RFC compliance, which removes most of the guesswork that existed before 2022.

Why the standard is short

The whole RFC is a small document, and that is the point. The grammar has only a handful of field names, and the entire matching algorithm reduces to a few sentences. When your file behaves unexpectedly, the cause is almost never a subtle detail of the standard. It is a rule you wrote too broadly, a group that does not apply to the crawler you think it does, or a file that is not being served the way you assume. The next three sections cover those three failure points in order.

Where the File Lives: Protocol, Host, Port, and the www vs Apex Rule

A robots.txt file is a UTF-8 plain-text file served at the root of an origin, so https://example.com/robots.txt, and its rules apply only to that exact host, protocol, and port. Google's introduction (2025) spells this out, and it is the source of a surprising amount of silent breakage. A file on example.com does not govern www.example.com, and a file served over HTTPS does not govern the HTTP origin, and a file on port 443 does not govern a non-standard port. Each of those is a different origin with its own file.

The www versus apex split

The most common failure is the www split. If your canonical domain is www.example.com but you only published a robots.txt at example.com/robots.txt, then www.example.com/robots.txt returns a 404 or a redirect, and Google applies the 404 rule we cover in the status code section. The fix is not to maintain two files. It is to pick one canonical host and make the other a 301 redirect to it, then serve a single robots.txt on the canonical host. The redirect of the non-canonical origin should point the whole origin, robots.txt included, at the canonical one, so both the file and every page resolve to one place.

Protocol and port

The same logic applies to protocol. If both HTTP and HTTPS are live on your domain, you should redirect HTTP to HTTPS at the server level, which also redirects the robots.txt file. A non-standard port is its own origin, which matters only if you expose content on one, and in that case that port needs its own file. In practice, keep one host, one scheme, and the default port, and redirect everything else. Then you only ever have to think about one robots.txt.

You can verify which origin you are actually serving with a single fetch. The robots.txt tester loads the file for the domain you give it and shows you the raw content and status, so a www versus apex mismatch shows up immediately as an empty or wrong file rather than a subtle crawl problem weeks later.

The Four Directives Google Supports: User-agent, Disallow, Allow, and Sitemap

Google officially supports four field names in robots.txt: user-agent, disallow, allow, and sitemap, plus the * wildcard, the $ end-of-URL anchor, and # comments, and it ignores directives it does not understand. Google's create-robots-txt documentation (2025) is the reference for that list. Everything else you have seen in other people's files, from crawl-delay to noindex to visit-time, is either an extension another bot honors or a directive Google skips entirely.

The directive reference

The table above is the whole vocabulary. There is no noindex line, no crawl-delay that Google will read, and no nofollow. If you need those behaviors, they live elsewhere: noindex goes in a meta tag or the X-Robots-Tag header, and crawl delay for Google is controlled through crawl settings in Search Console, not the file. The technical SEO guide covers the header and meta tag side in depth.

Wildcards, $ Anchors, and Comments: The Complete Syntax Reference

Three syntax features carry most of the real-world power in a robots.txt file, and they are easy to get wrong. The * wildcard matches any sequence of characters, including none, and it can appear anywhere in a path. The $ anchor, placed at the end of a path, means the match must end there, which is how you block an exact URL instead of everything that shares its prefix. The # character starts a comment that runs to the end of the line, and comments are for humans only, they never affect matching.

How matching actually reads

A Disallow or Allow value is matched as a prefix against the URL path, starting after the host. A rule of Disallow: /private matches /private, /private/, /private-notes, and /private/anything/else, because prefix matching does not care about word boundaries. That single fact explains most accidental blocks. To block only the exact directory, write Disallow: /private/ with the trailing slash, and to block only one exact file, add the anchor, like Disallow: /private/notes.html$.

Directive names are case-insensitive, but path values are matched case-sensitively, so Disallow: /Admin does not block /admin. The wildcard is the tool for matching patterns you cannot enumerate, like every URL that contains a query string (Disallow: /*?) or every PDF on the site (Disallow: /*.pdf$). Here is a complete annotated file that uses all of it.

# robots.txt for www.example.com
# Comments run to the end of the line and are ignored by crawlers.

# Default group: rules below apply to every crawler that has no
# more specific group elsewhere in the file.
User-agent: *
Disallow: /admin/
Disallow: /account/
Disallow: /checkout
Disallow: /search

# Block every URL that contains a query string, but leave /search
# itself crawlable. The $ anchor makes the second rule match only
# the exact path with nothing after it.
Disallow: /*?
Allow: /search$

# Block all PDF files anywhere on the site.
Disallow: /*.pdf$

# Point crawlers at the sitemap. This line is not tied to any group.
Sitemap: https://www.example.com/sitemap.xml

The wildcard and anchor are the difference between a surgical file and a blunt one, and both are supported by Google. Google's documentation (2025) lists * and $ alongside the four directives as supported syntax, so a pattern that reads correctly here will parse correctly in Googlebot. Before you rely on any pattern, paste the file and a test URL into the robots.txt tester and read the verdict it prints, because pattern mistakes are invisible until a page disappears from search.

How Google Picks the Winning Rule: Longest Match and Specificity, Worked

When several rules could apply to a single URL, Google does not apply them all and it does not apply the first one. It applies the longest, most specific match, and it applies the most specific user-agent group over the generic one. Google's create-robots-txt documentation (2025) states the rule in one sentence: the longest match wins, and the most specific group applies. Understanding it is the difference between a robots.txt that behaves and one that blocks pages you never meant to touch.

Groups do not stack

The first thing to unlearn is the idea that a crawler reads the * group and then layers its own group on top. It does not. A crawler picks the single most specific matching group and applies only that group's rules. If a Googlebot group exists, Googlebot ignores the * group entirely. That is why a file that blocks everything with User-agent: * and Disallow: /, but then adds a User-agent: Googlebot group with no Disallow lines, gives Googlebot full access to everything, because the empty Googlebot group overrides the blanket block. The * group only applies to crawlers with no more specific group.

Longest match wins

Within the group that applies, Google compares the matching rules and picks the longest one. Consider a group with Allow: /p and Disallow: /path and a URL of /path/file. Both match, but /path is longer than /p, so Disallow wins. Now add Allow: /path/file.html. For the URL /path/file.html, the Allow is now the longest match and wins, carving the exception back out. When an Allow and a Disallow match at exactly the same length, Allow wins, which is the tie-breaker the RFC and Google both use.

Worked examples

Walk each row against the algorithm and you will internalize it faster than reading the spec. The practical habit is to test rather than reason, because the interactions compound quickly once wildcards are involved. Every row in that table can be reproduced in seconds in the robots.txt tester, which applies the same longest-match logic Google uses, so you can paste your own rules and get the verdict per crawler before anything goes live.

Build Your First Robots.txt: A Step-by-Step Workflow

Most people start a robots.txt by copying someone else's and then wonder why parts of their site stopped being crawled. The correct order is the reverse: start permissive, then block specific things, and verify each block as you add it. The workflow below produces a file you understand, which matters more than a file that looks impressive.

Step one: inventory what should not be crawled

List the paths that add no value to search: internal search results, account and cart pages, staging, admin, thank-you pages, and anything that returns thin or duplicate content. Write the list before you write any rules, because the list is the specification and the file is just its translation.

Step two: start with a permissive default

Begin with a single User-agent: * group and no Disallow lines, or only the few obvious blocks. Every rule you add after that is a decision, and a minimal file is easier to audit than one with forty lines nobody remembers writing.

Step three: add one block at a time and test it

Add a Disallow for one path, then test a URL inside and outside that path in the robots.txt tester. The inside URL should come back blocked and the outside URL should come back allowed. If a rule blocks more than you intended, the tester shows you the exact winner, and that is the moment to fix the pattern before it ships.

Step four: add the AI crawler groups

Decide, per vendor, which AI crawlers may read the site. The matrix in the AI crawler section gives you the tokens and the effect of blocking each one. These go in their own groups so they do not interfere with the Googlebot rules.

Step five: declare the sitemap and ship

End with the Sitemap: line pointing at your canonical XML sitemap, then upload the file and verify it returns HTTP 200 as plain text at the root of your canonical origin. The next section covers that verification in full, because a correct file served from the wrong origin is still a broken file.

Validate Every Rule: Google's Robots.txt Tester and URL Inspection Test

Writing the file is only half the job. The other half is proving, before it goes live, that each rule does what you think it does. Google provides two tools for this in Search Console: the robots.txt tester, which parses your file and shows you which rules apply to a test URL for a given crawler, and the URL Inspection tool, which fetches a live URL and reports whether robots.txt is blocking it. Google's documentation (2025) describes both.

Test per crawler, not per file

The tester matters because a single file can behave completely differently for Googlebot and for GPTBot depending on the groups. Paste the file, select a user-agent, then test a URL you want blocked and a URL you want open. Do not test one URL and declare victory. The bugs live at the boundaries, exactly where a blanket rule and an exception meet, so test a URL inside each Disallow, a URL inside each Allow, and a URL right next to each boundary.

Run the live test after you deploy

The tester checks your intended file. The URL Inspection test checks the file Google is actually fetching from the live origin, which is a different thing and a more important one. After you deploy, run URL Inspection on a blocked page and confirm Google reports it as blocked, then run it on a page that should stay open and confirm it is not. This catches the whole class of problem where your robots.txt is correct but your CDN, server, or origin routing is serving something else, the topic of the edge interception section.

For a fast local check before touching Search Console, the robots.txt tester loads a live file for a domain and lets you test URLs against the longest-match rule immediately. Use it while you iterate, then confirm in Search Console before you trust the result, because Google's tester is the one that reflects what Google will actually do.

The Disallow vs Noindex Trap: Why Blocked Pages Still Get Indexed

This is the single most expensive misconception in SEO, so it gets its own section. A Disallow rule prevents crawling, not indexing. A URL blocked in robots.txt can still appear in Google's index when other pages link to it, because Google never fetches the blocked page and therefore never sees the noindex instruction that would remove it. Google's block-indexing documentation (2025) states the mechanism directly, and it produces the confusing Search Console status "Indexed, though blocked by robots.txt."

How the loop works

Imagine you want a page out of search, so you add Disallow: /old-page to robots.txt. Google stops crawling the page, but the page is still linked from your site and from others, so Google keeps it in the index with a fallback title and no snippet, because it cannot fetch the page to learn anything new. You have removed your ability to control the page while leaving it in search. The correct fix is the opposite: keep the page crawlable and put noindex on it, so Google fetches the page, reads the instruction, and drops it from the index. The noindex directive belongs in a <meta name="robots" content="noindex"> tag or an X-Robots-Tag header, never in robots.txt. Google's block-indexing documentation (2025) is explicit that robots.txt cannot be used to prevent indexing.

Disallow versus noindex at a glance

The third row is the trap people fall into most often. They combine the two mechanisms, thinking more protection is better, and the Disallow silently cancels the noindex because the page can no longer be fetched to read it. Pick one mechanism for one goal. To stop crawling, use Disallow. To stop indexing, use noindex on a crawlable page. The on-page SEO guide covers the noindex meta tag and how to audit it on a live page.

HTTP Status Codes and Redirects: What 4xx, 5xx, and 30x Tell Crawlers

The status code your server returns for /robots.txt is itself a directive, and it overrides whatever is inside the file. A 4xx status tells Google there are no restrictions, so it may crawl everything. A 5xx status is treated as a temporary stop, so Google pauses crawling and retries with backoff. A 30x redirect makes the crawler follow the redirect, up to a limit. Google's robots.txt introduction (2025) documents the 4xx and 5xx behavior explicitly.

The status table

The two rows that surprise people are the 4xx and the 5xx. A missing robots.txt is not a refusal, it is an invitation: Google reads a 404 as "no rules, crawl freely." And a persistent 5xx is worse than a wrong rule, because instead of applying the wrong rule it stops crawling entirely and retries later. If your site is getting crawled far less than expected and your robots.txt returns a 500, you have found the cause.

Redirect handling

A redirect on robots.txt is normal when you canonicalize www to apex or HTTP to HTTPS, and a single 301 to the canonical file is the clean way to do it. What you want to avoid is a chain of redirects, or a redirect to a page that returns HTML rather than the plain text file, because each hop is another chance for the file to be dropped. Fetch your own file and follow the hops: it should land on exactly one 200 with Content-Type: text/plain, and the robots.txt tester shows the status and the final file content in one pass.

Declaring Sitemaps: The Cross-Submission Shortcut That Works

The Sitemap: directive in robots.txt is a discovery shortcut that points crawlers at your XML sitemap files without you having to submit them one by one in Search Console. It is defined by the sitemaps.org protocol (version 0.9) and is honored by Google and Bing, which is why it is the most reliable cross-submission mechanism you have. Declare it in every robots.txt you publish, on every origin, and the crawlers will find your sitemap even when nobody remembers the Search Console property. What belongs inside the file itself is the subject of the XML sitemaps guide.

The line is simple and lives outside any user-agent group, because it is not a crawl rule, it is a pointer. It must be an absolute URL, and it should point at your canonical sitemap. You can declare more than one sitemap by adding more Sitemap: lines, one per file, which is how you handle a large site split across a sitemap index.

The part people get wrong is the contents of the sitemap they declare. Pointing crawlers at a sitemap full of redirected, noindexed, or robots-blocked URLs wastes crawl budget and blurs the signal the sitemap exists to send. Before you declare a sitemap, run it through the sitemap checker and confirm every listed URL is canonical, indexable, and returns 200. A clean sitemap declared in robots.txt is the cheapest crawl-quality win available, and a dirty one actively works against you. The technical SEO guide covers sitemap hygiene and the 50,000 URL limit in more detail.

The 2026 AI Crawler Matrix: GPTBot, OAI-SearchBot, ChatGPT-User, and More

Which AI crawlers may read your site is now a first-class decision alongside Googlebot, and the vendors have made it controllable per bot through robots.txt. The mistake most sites make is a blanket block that also excludes the crawlers that make a site citable in AI answers, or a blanket allow that feeds model training without the owner realizing it. The right approach is to decide per vendor and per purpose, because the bots feed different products.

The three purposes

AI crawlers split into three groups. Training crawlers such as GPTBot and ClaudeBot add your content to future model training. Search crawlers such as OAI-SearchBot and PerplexityBot index you for AI search products and their citations. Assistant fetchers such as ChatGPT-User and Amazonbot open your pages when a person asks an assistant to read a specific URL. Blocking the first group opts you out of training. Blocking the second and third stops assistants from citing or summarizing you. The generative-search visibility matters because you cannot be cited by an engine whose crawler you have blocked, and the AI visibility checker helps you see which engines can reach your content today, and the AI search visibility guide walks the full citation pipeline.

The 2026 matrix

The token list and purposes come from the vendors' own documentation. OpenAI describes its three independently controllable crawlers in its bots documentation (2025), and Google publishes a ready-made robots.txt block for AI crawlers including GPTBot, CCBot, Google-Extended, and PerplexityBot in its useful robots.txt rules reference (2025). Anthropic documents ClaudeBot (2025), Amazon documents Amazonbot (2025), and Perplexity documents its crawlers in its crawler reference (2025). The two Google tokens deserve their own section, because the distinction between them is the decision most publishers get wrong.

Google-Extended vs Googlebot: Opt Out of AI Training Without Losing Rankings

Google uses two different tokens to give publishers independent control, and confusing them is costly. Googlebot controls Google Search indexing, and blocking it removes you from Search. Google-Extended is a separate, standalone token that lets you opt out of having your content used to train Google's AI models such as Gemini, and blocking it does not affect your Search indexing or ranking. Google's crawler overview (2025) states this separation explicitly.

The practical implication is that you can, and often should, treat the two differently. Many publishers want to stay in Google Search while declining to donate their content to Gemini training, and Google-Extended exists exactly for that. The block is a single group:

User-agent: Google-Extended
Disallow: /

That two-line group opts your site out of Google's AI training crawler while leaving Googlebot untouched, so Search traffic is unaffected. The reverse is also possible: a site that is happy to feed Gemini training but does not want Search traffic is rare, but the tokens are independent and you can choose per site.

Why this matters for AI visibility

The decision is not purely defensive. Blocking Google-Extended and GPTBot means your content will not be in the training data of the largest frontier models, which is what most publishers want by default. But blocking the search and assistant crawlers, like OAI-SearchBot and ChatGPT-User, means you stop appearing in AI answers and citations, which is a traffic decision, not a privacy one. Separate the training opt-out from the search opt-in, and use the AI visibility checker to confirm which engines can still reach your content after you make the change.

Copy-Paste Robots.txt Blocks for Every AI Crawler

The matrix above is the decision layer. This section is the implementation layer: one complete, copy-paste block per vendor that you can append to your existing robots.txt. Each block is a separate user-agent group, so each vendor's rules stay independent and none of them interfere with your Googlebot rules. Google's own useful robots.txt rules reference (2025) publishes a similar ready-made block, and the version below adds the vendor-specific tokens for ClaudeBot, Bytespider, and Amazonbot so you can copy it once and be done.

# AI crawler blocks: each group is independent and only affects that vendor.

# OpenAI training
User-agent: GPTBot
Disallow: /

# OpenAI search (ChatGPT search results and citations)
User-agent: OAI-SearchBot
Disallow: /

# OpenAI assistant (user-prompted page fetches)
User-agent: ChatGPT-User
Disallow: /

# Anthropic training
User-agent: ClaudeBot
Disallow: /

# Perplexity search
User-agent: PerplexityBot
Disallow: /

# Common Crawl corpus (feeds many models)
User-agent: CCBot
Disallow: /

# ByteDance and TikTok AI crawls
User-agent: Bytespider
Disallow: /

# Amazon assistant services
User-agent: Amazonbot
Disallow: /

Before you paste this, decide which blocks you actually want, because a blanket paste of all eight opts you out of AI search citations as well as training. If your goal is training opt-out only, delete the OAI-SearchBot, ChatGPT-User, PerplexityBot, and Amazonbot groups and keep the training crawlers. If you also want to stop appearing in AI answers, keep them. The block is a menu, not a mandate, and each line is independently removable.

After you append the block, test it the same way you would any other rule. Paste the file into the robots.txt tester, select GPTBot, and confirm a URL comes back blocked, then select Googlebot and confirm the same URL is still allowed. That one check catches the most dangerous bug in this whole section: accidentally applying an AI block to Googlebot and vanishing from Search while you thought you were only opting out of training.

When Robots.txt Is Not Enough: WAF, CDN, and Server-Side Enforcement

robots.txt is a request, and requests can be ignored. The file is a voluntary convention, not a security mechanism, and Google's documentation (2025) says outright that malicious bots and scrapers may ignore it entirely. If you actually need to stop a scraper, a credential-stuffing bot, or an aggressive crawler that ignores the protocol, robots.txt is the wrong layer, and relying on it is how sites leak data while believing they are protected.

Real enforcement lives in three places. A web application firewall, or WAF, inspects traffic by signature and behavior and can block known bad-bot user-agents and IP ranges before the request reaches your server. A CDN edge can apply rate limits, challenge suspicious requests, and drop traffic from datacenter IP ranges, which is where most scraping originates. And your server itself can block or throttle specific user-agents and IPs in its configuration. All three operate on the request itself, which is what makes them enforcement where robots.txt is only a suggestion.

Use both layers for different jobs

The correct mental model is two layers. robots.txt handles the polite crawlers you have a relationship with, the search engines and AI vendors that read the file and respect it. The WAF, CDN, and server config handle the actors that ignore the file, using signatures and rate limits that do not depend on the bot's willingness to cooperate. Do not expect robots.txt to stop a scraper, and do not expect a WAF to replace the fine-grained per-path control robots.txt gives you over Googlebot. They solve different problems and you need both. The technical SEO guide covers the server and header side of enforcement in more depth.

Edge Interception: Why Cloudflare, Fastly, or Akamai Can Shadow Your File

A subtle and common failure is that your robots.txt is correct at the origin but wrong at the edge. Cloudflare, Fastly, Akamai, and similar services sit between the crawler and your server, and any one of them can cache, rewrite, or shadow the file, so the crawler sees something different from what you published. The symptom is usually a correct file on disk and a traffic change you cannot explain, which is exactly the kind of silent problem that takes weeks to diagnose.

The three ways edges break the file

First, a caching rule can serve a stale or empty version of /robots.txt long after you changed it. Second, a bot-management rule can intercept requests from a user-agent you want to allow, such as Googlebot or GPTBot, and return a challenge page or a block, which the crawler reads as the robots.txt content and misparses. Third, a page-rule or worker can rewrite the path, so /robots.txt returns your HTML 404 page or a redirect to the homepage, which the crawler interprets according to the status code rules from the HTTP section.

Diagnosing a shadowed file

The diagnosis is to compare what the edge serves with what the origin serves. Fetch /robots.txt through the public domain and read the raw body and status, then fetch it directly from your origin IP with a Host header override and compare. If they differ, the edge is shadowing the file. The robots.txt tester fetches the public file, so it shows you the edge version, which is the version crawlers actually see. If the tester shows a 404, an HTML page, or stale rules, you have found the shadow even before you look at the origin. Then fix the cache key, the bot rule, or the page rule so the edge passes /robots.txt through untouched, and re-test until the public file matches your intent.

The 10 Most Common Robots.txt Mistakes and How to Fix Each One

Nearly every broken robots.txt on the web is a variation on a small set of mistakes. The list below is remediation order, from the mistake that does the most damage to the one that just wastes a line. Each has a concrete fix you can apply and verify with the tools already covered.

1. Blocking Googlebot while trying to opt out of AI training

Pasting a blanket AI block that includes a User-agent: * Disallow, or confusing Google-Extended with Googlebot, removes the site from Search. Fix it by scoping the AI blocks to their own groups and leaving the Googlebot group permissive, then verify in the robots.txt tester that Googlebot still has access.

2. Using Disallow to keep a page out of the index

The page stays indexed from links and you lose control of it. Fix it by removing the Disallow, keeping the page crawlable, and adding noindex, as detailed in the Disallow versus noindex section.

3. A 404 or 5xx on robots.txt

A 404 means crawl everything and a 5xx means crawl nothing for a while. Fix it by serving a real 200 file at the root of your canonical origin and checking the status when you fetch it in the robots.txt tester.

4. The www versus apex split

The file lives on one host but not the other, so half your origin has no rules. Fix it by 301-redirecting the non-canonical host to the canonical one, file included.

5. Prefix rules that match more than intended

Disallow: /blog also blocks /blogger. Fix it with a trailing slash or a $ anchor, then test a URL on each side of the boundary.

6. Forgetting that groups do not stack

An empty Googlebot group overrides a blanket * block and gives Googlebot full access. Fix it by putting the intended rules inside the specific group, not assuming it inherits the generic ones.

7. Wasting lines on crawl-delay and noindex

Google ignores both, so they do nothing for Googlebot. Fix it by removing them and using the real mechanisms: Search Console crawl settings for rate, and meta robots for noindex.

8. Declaring a dirty sitemap

A sitemap full of redirected and blocked URLs wastes crawl budget. Fix it by cleaning the sitemap with the sitemap checker before you declare it.

9. Never testing the file you shipped

The file on disk differs from the file the edge serves. Fix it by testing live in the robots.txt tester and in Search Console's URL Inspection after every deploy.

10. Treating robots.txt as a security wall

Scrapers ignore it and the content leaks. Fix it by keeping real enforcement in the WAF and CDN, covered in the enforcement section, and using robots.txt only for cooperative crawlers.

Work the list top to bottom and most sites will find at least three of these live right now. Each fix is small, but the compounding effect on crawl behavior and indexation is not, which is why a well-formed robots.txt is worth more than most people assume.