Module 3: Technical Foundations
Crawl and indexation, sitemaps, robots.txt and the HTTP layer, without the jargon wall.
Search engines do not see your site the way a visitor does. Before a page can rank, a crawler has to find it, fetch it, and decide it is worth keeping. That pipeline from first visit to indexed page is what this module covers. It is the part of SEO that makes everything else possible.
This module walks three core files and the signals around them. robots.txt tells crawlers which parts of your site they may fetch. An XML sitemap lists the URLs you want found. And the HTTP layer, made up of status codes and headers, tells a crawler what each response means.
You will learn what crawling and indexing actually are, and why they are not the same thing. You will learn the limits a sitemap has to respect, why the lastmod date matters more than most people think, and how one misplaced line in robots.txt can hide a whole section of your site.
By the end you can read these files, spot the mistakes that slow a crawl, and fix them with confidence. Each quiz question tests one core fact, so the quiz doubles as a review of the details that matter.
What you will learn
- Explain the difference between crawling and indexing and why a robots.txt block does not remove a page from search results.
- Write a robots.txt file that uses Disallow, Allow, and Sitemap correctly, including the longest-match rule.
- Build and validate an XML sitemap that stays under the 50,000 URL and 50 MB limits and keeps lastmod honest.
- Read HTTP status codes and headers to spot soft 404s, redirect chains, and indexing directives.
- Choose which AI crawlers to allow or block without removing a site from Google Search.
Key takeaways
- robots.txt governs crawling, not indexing, and a blocked page can still appear in search results.
- Google applies the longest, most specific match when two robots.txt rules collide, and Allow wins ties.
- A single sitemap holds at most 50,000 URLs and 50 MB uncompressed, and Google ignores priority and changefreq.
- lastmod is a binary change flag, so stamping a fake fresh date destroys the only signal Google reads.
- A page that returns 200 while saying it is not found is a soft 404 and wastes crawl budget.
- Block Google-Extended to opt out of AI training, but never confuse it with Googlebot, which controls Search.
Reading for this module
Module 3 quiz
Ten questions. Pass mark is 7 of 10. Retake as often as you like; your best score counts.