Meta Robots and X-Robots-Tag: Directives, Header vs Tag, Mistakes

Meta robots and X-Robots-Tag share one directive set — noindex, nofollow, nosnippet, max-snippet, unavailable_after and more. Both are seen only when a page is crawled, so a robots.txt-blocked page never has its noindex read.

Meta robots and X-Robots-Tag carry the same directive set and do the same job: they control what a crawler does with a page once it has fetched it. There is exactly one structural difference — the meta tag lives in the HTML <head> and works only for HTML, while the header works for any file type and can be applied site-wide. The single fact that causes the most damage: both are seen only when a page is crawled, so a page blocked in robots.txt never has its noindex read.

The shared directive set

Both mechanisms accept the same values. The ones that matter (handbook Ch.2):

Directive	Meaning
`all`	No restrictions — the default when nothing is set
`noindex`	Exclude the page from search results
`nofollow`	Do not follow the links on this page
`none`	Equivalent to `noindex, nofollow`
`nosnippet`	No text or video snippet — and it also blocks the page from being used as input to AI Overviews and AI Mode
`max-snippet:[n]`	Cap snippet length in characters; `0` means nosnippet, `-1` lets Google decide
`max-image-preview:[none\|standard\|large]`	Image preview size (default `standard`)
`max-video-preview:[n]`	Max preview seconds; `0` = static image, `-1` = no limit
`noimageindex`	Do not index images on the page
`unavailable_after:[date]`	Drop the page from results after the given date
`indexifembedded`	Index iframe-embedded content despite `noindex` (only paired with `noindex`)

Two notes on behavior. noarchive is effectively a no-op today. And conflicts resolve to the most restrictive rule: a crawler obeys the sum of all applicable negative directives across every tag or header targeting it, so a noindex anywhere in the applicable set wins.

<meta name="robots" content="noindex, nofollow">
<meta name="robots" content="max-snippet:-1, max-image-preview:large, max-video-preview:-1">

You can also target a specific crawler by name — <meta name="googlebot" content="noindex"> — which applies only to that bot, leaving the generic robots rule for the rest.

Meta tag versus HTTP header

The choice between the two comes down to file type and scope.

Meta robots goes in the <head> of an HTML document. It is the natural place for per-page directives on normal pages, and it is easy to read and audit in the source.

X-Robots-Tag is an HTTP response header. Because it lives in the response rather than the body, it works for non-HTML resources — PDFs, images, feeds — that have no <head> to put a meta tag in. It is also the right tool for applying a directive at scale through server or CDN configuration, using a pattern match.

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

There is no ranking difference between the two. Use the header for non-HTML resources or pattern-based rules; use the meta tag for per-page HTML control. A noindex delivered by header is easy to set accidentally at the server level and invisible in page source, which is why Crawlinx surfaces it as directives.noindex_via_x_robots. It likewise flags a nosnippet directive as directives.nosnippet, because it silently removes the page from AI Overview and AI Mode inputs.

robots.txt versus robots directives: crawl versus index

This is the distinction that trips up the most sites, and it is worth stating plainly. robots.txt controls crawling — whether a bot may fetch a URL at all. Meta robots and X-Robots-Tag control indexing and serving — what a bot does with a page it has already fetched. They operate at different stages, and they do not substitute for each other.

The consequence is the number-one indexing pitfall: a Disallow in robots.txt and a noindex on the same URL are mutually defeating. If robots.txt blocks the URL, Google cannot crawl it, so it never sees the noindex in the head or header. The page can then remain indexed URL-only — appearing in results as a bare link with no snippet — because Google can index a URL it has never fetched when other pages link to it. You asked for the page to be removed and instead guaranteed it can never be.

The correct way to deindex a page: keep it crawlable and serve noindex. Let Google fetch the page, read the directive, and drop it. Only after the page has been recrawled and deindexed is it safe to block it in robots.txt if you also want to stop wasting crawl budget on it. Crawlinx flags a URL blocked from crawling as robots.blocked, and separately warns when robots.txt blocks the CSS or JS a page needs to render as robots.blocks_css_js. The full mechanics of the crawl side are in our robots.txt guide.

For a page that must come out of the index urgently, noindex is permanent but requires a recrawl to take effect. The Search Console Removals tool is fast — roughly a day — but temporary, expiring in about six months. For urgent and permanent removal, use both.

Common mistakes

Blocking a page in robots.txt to deindex it. As above, this prevents Google from ever seeing the noindex. The page can stay indexed URL-only.
Canonicalizing to a noindex page. Pointing a canonical at a URL that says "do not index me" gives Google two contradictory instructions, and the consolidation you wanted does not happen. Crawlinx flags this as canonical.to_noindex.
A leftover site-wide noindex from staging. A noindex applied across a staging environment that ships to production deindexes the whole site. A header-delivered one is especially easy to miss because it is invisible in page source.
Relying on noindex in robots.txt. Google removed support for the noindex directive inside robots.txt on 2019-09-01. Any robots.txt still using it is doing nothing.
Expecting nosnippet to be free. It suppresses the snippet, but it also removes the page as an input to AI Overviews and AI Mode — a cost most people do not intend.
Conflicting directives across tag and header. When both a meta tag and a header apply, the most restrictive combination wins. A permissive meta tag will not override a restrictive header.

How Crawlinx detects these

Crawlinx reads both the meta robots tag in the <head> and the X-Robots-Tag response header on every crawled URL, and reconciles them into the effective directive set. It flags a header-delivered noindex you might not expect (directives.noindex_via_x_robots), a nosnippet that removes AI-answer eligibility (directives.nosnippet), URLs blocked from crawling (robots.blocked), render-critical assets blocked in robots.txt (robots.blocks_css_js), and canonicals pointing at a noindex target (canonical.to_noindex). See where indexing controls sit in the full technical SEO audit checklist, and how blocked and noindexed pages waste crawl demand in the crawl budget guide.

Takeaway

Meta robots and X-Robots-Tag share one directive set; pick the header for non-HTML files or pattern-based rules, the tag for per-page HTML control. Remember the split: robots.txt governs crawling, robots directives govern indexing. To remove a page from results, keep it crawlable and serve noindex — never block it in robots.txt and expect the noindex to be read.

Robots.txt Guide: How Robots.txt Works Technical SEO Audit Checklist Crawl Budget: What It Is and How to Manage It directives.noindex_via_x_robots directives.nosnippet robots.blocked robots.blocks_css_js canonical.to_noindex

Audit your own site — free

77 checks, internal PageRank, render-diff. No signup, results in ~30s.

Scan your site →