Robots.txt Guide: How It Works and How to Get It Right

Robots.txt is a plain-text file at your host root that tells crawlers which URLs they may fetch. It manages crawl traffic — it does not keep pages out of the index. It caps at 500 KiB, must return HTTP 200, and must never block the CSS or JavaScript your pages need to render. Everything else is syntax and precedence.

What robots.txt is for

The single most important fact about robots.txt: its purpose is to manage crawl traffic, not to hide pages. A URL you disallow can still be indexed URL-only if another site links to it, because Google can index a URL it has never fetched. To keep a page out of the index you must let Google crawl it and serve a noindex directive, or protect it with a password. This distinction — crawl control versus index control — is where most robots.txt mistakes begin.

Robots.txt lives at exactly /robots.txt at the host root, one file per host. Its scope is a single protocol, host, and port combination: https://www.example.com/robots.txt governs https://www.example.com only, not the http:// version, not a subdomain, not another port. The file is UTF-8 (a leading BOM is ignored).

Syntax and the four supported directives

Google supports exactly four fields: user-agent, disallow, allow, and sitemap. Field names are case-insensitive, but path values are case-sensitive — /Page and /page are different rules.

User-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Pattern matching uses two wildcards: * matches zero or more characters, and $ anchors the end of the URL ($ does not apply to the sitemap line). Common patterns:

Disallow: /*?          # any URL with a query string (faceted params)
Disallow: /calendar/   # a directory
Disallow: /*.pdf$      # all PDFs

Google does not support crawl-delay, nofollow, or noindex inside robots.txt — the noindex directive was removed on 2019-09-01. A robots.txt that relies on any of these is silently doing nothing. Real-world audits back this up: robots.txt files collapse to allow, disallow, and user-agent in practice, and exotic directives are wasted effort.

Precedence: how conflicts resolve

Two rules matter here.

Longest matching path wins. Between Allow: /folder/page and Disallow: /folder/, the more specific Allow rule takes the URL because its matching path is longer.

On a tie, the least restrictive rule wins — that is, Allow beats Disallow when both match the same number of characters.

A crawler obeys exactly one group: the most specific user-agent match. The * group is never merged into a named bot's group. If you write a User-agent: Googlebot block, Googlebot follows only that block and ignores the User-agent: * block entirely — a frequent source of surprise when a narrow bot rule accidentally opens up paths the wildcard group blocked.

The sitemap directive

A Sitemap: line declares your sitemap's absolute URL. It is one of the two legitimate ways to tell Google where your sitemap lives (the other is Search Console). The old unauthenticated sitemap ping endpoint was deprecated in June 2023 and now returns 404, so a robots.txt Sitemap: line is worth keeping current.

The 500KB limit and why the file must return 200

Google reads only the first 500 KiB of robots.txt; everything past that byte is ignored. If your rules sprawl past that limit, the tail-end disallows silently stop applying. Keep the file lean.

The status code robots.txt returns matters more than most people realize, because failure handling differs from normal page rules:

• 2xx — parsed and used.
• 4xx (except 429) — treated as no robots.txt exists, so Google crawls everything.
• 429, 5xx, DNS or timeout — Google stops crawling the entire site for roughly the first 12 hours, then falls back to the last cached copy for up to 30 days.

The pitfall: a 5xx on robots.txt halts crawling of your whole site short-term, which is far worse than a 404. Never serve 5xx or 429 on robots.txt as a throttle. Google also caches robots.txt for up to 24 hours by default (it honors Cache-Control: max-age), so a fix is not instant. If your site has no robots.txt at all, that is not fatal — Google reads it as "crawl everything" — but a missing file is a missed chance to declare your sitemap and contain crawl waste. Crawlinx flags this as robots.txt_missing.

Never block CSS or JS

This is the one robots.txt mistake that can silently deindex working pages. Google indexes the rendered DOM, and its Web Rendering Service fetches your CSS and JavaScript the way a browser does. If robots.txt disallows those assets, the renderer produces a broken or blank page and the content that depends on them can vanish from the index. The failure hides in plain sight: the HTML still returns 200 and looks fine in your browser, because your browser ignores robots.txt while Google does not.

This most often hits sites that reflexively disallow a build or asset directory — /static/, /assets/, /_next/, /wp-includes/ — without noticing those paths serve render-critical files. See the full mechanics in our JavaScript SEO and rendering guide, and Crawlinx flags it as robots.blocks_css_js.

Common mistakes

• Using robots.txt to deindex a page. It only blocks crawling; the URL can stay indexed. Use noindex on a crawlable page instead. Worse, Disallow + noindex on the same URL are mutually defeating — Google cannot crawl the page to see the noindex.
• Blocking CSS or JS, breaking rendering (above).
• Blocking AI search crawlers by accident. Disallowing bots like OAI-SearchBot or PerplexityBot removes you from AI answer citations; Crawlinx flags this as geo.ai_bots_blocked.
• Serving 5xx or 429 on the file, halting site-wide crawling.
• Exceeding 500 KiB so later rules are ignored.
• Assuming the * group applies to a named bot — it does not.

How Crawlinx helps

Crawlinx parses your robots.txt with Google's own matching rules and flags render-critical assets that are disallowed (robots.blocks_css_js), a missing file (robots.txt_missing), URLs blocked from crawling (robots.blocked), and AI search bots blocked from citation (geo.ai_bots_blocked).

Takeaway

Robots.txt controls crawling, never indexing. Keep it under 500 KiB, make sure it returns 200 (never 5xx), let Googlebot fetch your CSS and JS, and reach for noindex — not Disallow — when your real goal is to keep a page out of results.