Handbook/Ch. 2
Handbook · Chapter 2

Crawling & Indexing

How Googlebot crawls, how crawl budget works, how each HTTP status code is treated, and how robots.txt, meta robots, canonicalization, and sitemaps interact. The #1 indexing pitfall: robots.txt disallow plus noindex defeat each other.

How Google crawls

Crawler UA tokens (for robots.txt / X-Robots-Tag):

Crawler Token Purpose
Googlebot (Desktop + Smartphone) Googlebot Main index (Search, Images, Video, News, Discover)
Googlebot-Image/-Video/-News own tokens (fall back to Googlebot) Media/News
Storebot-Google Storebot-Google Shopping
Google-InspectionTool Google-InspectionTool Rich Results Test, URL Inspection
GoogleOther GoogleOther Generic non-index crawls
Google-Extended Google-Extended Gemini training opt-out token (does NOT affect Search)
AdsBot-Google AdsBot-Google Ad quality; ignores the * group — name it explicitly

Crawl budget

Crawl budget = crawl capacity limit + crawl demand. Capacity rises with fast/healthy responses, drops on slowdowns, 5xx, or 429. Demand is driven by perceived inventory (dupes waste it — your biggest lever), URL popularity, staleness, site moves.

Who needs to manage it (Google's thresholds):

Situation Manage?
1,000,000+ pages, changing ~weekly Yes
10,000+ pages, changing daily Yes
Large share of "Discovered – currently not indexed" Yes
Smaller sites / crawled same day No — keep sitemaps current

DO: consolidate duplicates; robots.txt-block genuinely useless URLs; return 404/410 for gone pages; fix soft 404s; fresh sitemaps with accurate <lastmod>; short redirect chains; fast server. DON'T: rely on noindex to save budget (page still fetched every time); use 401/403/5xx to throttle. Crawl rate is not a ranking signal.

2026 crawl-waste reality: waste is dominated by faceted navigation, action parameters, and irrelevant URL parameters — classic technical hygiene is the highest-leverage crawl fix. Illyes: "it's not crawling that eats resources, it's indexing and what you're doing with the data."

HTTP status codes — how Google treats each

Status Behavior
200 Sent to indexing (not guaranteed)
201/202 Waits limited time, then proceeds
204 No content processed
301 / 308 Followed; strong canonical signal → target
302 / 303 / 307 Followed; weak (temporary); source may stay indexed
304 Unchanged; indexing unaffected
4xx (400/401/403/404/410/411, not 429) Content unused; previously-indexed URLs removed; crawl frequency decays
404 vs 410 Same removal; 410 (Gone) may drop slightly faster
429 Treated as a server error, not normal 4xx — crawlers slow
5xx (500/502/503) Ignored; crawlers slow; indexed URLs retained then dropped if persistent. Use 503 (+ optional Retry-After) for maintenance
Network/DNS/timeout Treated as 5xx

robots.txt

Purpose: manage crawl traffic — NOT to hide pages. A disallowed URL can still be indexed URL-only if linked externally. To keep a page out of the index use noindex or a password.

Property Value
Location Exactly /robots.txt at host root; one per host
Scope Exact protocol + host + port only
Encoding UTF-8 (leading BOM ignored)
Size limit 500 KiB (rest ignored)
Caching Up to 24 h default; honors Cache-Control: max-age
Fields Case-insensitive names; path values case-sensitive

Supported: user-agent, disallow, allow, sitemap. NOT supported by Google: crawl-delay, nofollow, and noindex (removed 2019-09-01). Matching: * = 0+ chars, $ = end-of-URL (not on sitemap). Longest matching path wins; on a tie, least-restrictive (allow) wins. A crawler obeys exactly one group (most specific UA match); * is never merged into a named UA's group.

robots.txt fetch-failure handling (differs from page rules — critical):

Status Effect
2xx Parse and use
3xx Follows ≥5 hops then treats as 404
4xx (not 429) Treated as no robots.txt → crawl everything
429 / 5xx / DNS / timeout Stops crawling the whole site first ~12 h; uses last cached copy up to 30 days; after 30 days if reachable → no restrictions, else stop

Pitfall: a 5xx on robots.txt halts crawling of the entire site short-term — far worse than a 404 (which means "crawl all"). Never serve 5xx/429 on robots.txt as a throttle.

2026 note: real-world robots.txt collapses to allow/disallow/user-agent; 84.9% return 200, ~13% return 404. Exotic directives are wasted effort. Make sure yours returns 200, not 404.

# Block one bot, allow the rest, declare sitemap
User-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

# Common patterns
Disallow: /*?          # any query string (faceted params)
Disallow: /calendar/   # a directory
Disallow: /*.pdf$      # all PDFs

Meta robots & X-Robots-Tag

Same directive set. Meta robots = HTML <head> only. X-Robots-Tag header = any file type + site-wide/regex. Both are seen only when the page is crawled — a robots.txt-blocked page never has its noindex read.

Directive Meaning
all No restrictions (default)
noindex Exclude from results
nofollow Don't follow page links
none = noindex, nofollow
nosnippet No text/video snippet; also blocks use as input to AI Overviews/AI Mode
max-snippet:[n] Max snippet chars; 0=nosnippet, -1=Google decides
max-image-preview:[none|standard|large] Image preview size (default standard)
max-video-preview:[n] Max preview seconds; 0=static, -1=no limit
noimageindex Don't index images on the page
notranslate No translation offer
indexifembedded Index iframe-embedded content despite noindex (only paired with noindex)
unavailable_after:[date] Drop after date

noarchive is effectively a no-op now. Conflicts resolve to the most restrictive. A crawler obeys the sum of all applicable negative rules across tags targeting it.

<meta name="robots" content="noindex, nofollow">
<meta name="robots" content="max-snippet:-1, max-image-preview:large, max-video-preview:-1">

<Files ~ "\.pdf$"> Header set X-Robots-Tag "noindex, nofollow" </Files>

<p>Shown <span data-nosnippet>not in snippet</span>.</p>   <!-- span/div/section only -->

The #1 indexing pitfall: Disallow: + noindex on the same URL are mutually defeating — Google can't crawl the page to see the noindex, so it may stay indexed URL-only. To deindex: keep the page crawlable and serve noindex.

noindex vs Removals tool: noindex is permanent but needs a recrawl. The Search Console Removals tool is fast (~1 day) but temporary (~6 months). Urgent-and-permanent = use both.

Canonicalization

Choosing the representative URL among duplicates. Declaring a canonical is a hint, not a rule.

Signals (strongest → weakest): 301/permanent redirect → rel="canonical" (link or HTTP header) → HTTPS-over-HTTP / hreflang clusters / consistent internal links → sitemap inclusion (weak). Methods stack.

<link rel="canonical" href="https://example.com/dresses/green" />

Link: <https://www.example.com/paper.pdf>; rel="canonical"    # for non-HTML

Rules: absolute URLs; one canonical per page; target must be a live 200, not noindex/robots-blocked, no fragment. Cross-domain canonical supported. Use self-referencing canonicals. Pitfalls: canonicalizing to a redirected/noindexed/blocked page; conflicting canonicals; canonicalizing paginated page 2+ to page 1 (wrong); JS changing canonical after render; HTTPS→HTTP redirects making Google prefer HTTP. Malformed HTML can silently break rel=canonical.

Sitemaps

Property Limit
URLs per sitemap 50,000
Uncompressed size 50 MB
Sitemap index 50,000 sitemaps / 50 MB (gzip supported)

Formats: XML (full), RSS 2.0 / Atom 1.0 (recent URLs), plain text. <changefreq> and <priority> are ignored by Google. <lastmod> (W3C datetime) used only if consistently accurate. Submit via Search Console / API sitemaps.submit, or a Sitemap: line in robots.txt. The unauthenticated ping endpoint is deprecated (June 2023, returns 404) — remove any code calling it.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://www.example.com/foo.html</loc><lastmod>2026-06-04</lastmod></url>
</urlset>

Specialized sitemaps (video, image, news, hreflang) are covered in Chapters 7 and 8.

Pagination & faceted navigation

rel="next"/"prev" deprecated (2019-03-21) — Google no longer uses them. Current guidance: unique URL per page (?page=n); real <a href> links; self-referencing canonical on each page (page 2 → page 2, never → page 1); consistent titles. DON'T use fragments (#page=2) or JS buttons ("Google's crawlers don't click buttons"). Infinite scroll/load-more: back each chunk with a persistent unique URL updated via History API pushState; a fast "view-all" page is an acceptable alternative.

Faceted navigation — #1 reported source of overcrawl:

Redirects

Type Passes canonical signal Use
301 / 308 Yes (permanent, strongest) Permanent moves
302 / 303 / 307 No (temporary) Temporary; source may stay indexed
Meta refresh 0s Treated permanent
Meta refresh >0s Treated temporary
JS location Permanent if executed Risky — render may fail

Reliability order: server-side > meta refresh > JS > manual. Keep chains ≤10 hops. Pitfalls: loops; redirecting all traffic to homepage → soft 404; JS redirects missed on render failure.

Removals, IndexNow & Indexing API

Search Console essentials

Verify ownership → check indexing → submit sitemap → monitor Performance.

Sources

  1. Google Search Central — Large site owner's guide to managing crawl budget
  2. Google Search Central — Introduction to robots.txt
  3. Google Search Central — Consolidate duplicate URLs (canonicalization)
  4. sitemaps.org — Sitemap protocol
  5. Google Search Central — HTTP status codes and network errors
Related
Robots.txt for SEO: Syntax, Rules & Common Mistakes Canonical Tags: How rel=canonical Fixes Duplicates XML Sitemaps: Format, Limits & Common Mistakes HTTP Status Codes for SEO: 2xx-5xx Explained Crawl Budget: What It Is and How to Manage It Meta Robots & X-Robots-Tag: noindex/nofollow Guide canonical.non_self canonical.missing sitemap.orphan_not_listed redirect.chain

Audit your own site — free

156 checks, internal PageRank, render-diff. No signup, results in ~30s.

Scan your site →