noindex via X-Robots-Tag header: catch the block you can't see in source

What it means

A page returns an X-Robots-Tag: noindex HTTP response header. This blocks indexation exactly like <meta name="robots" content="noindex">, but at the network layer before the HTML is parsed — and it is invisible in page source. It may be intentional, but is frequently accidental.

Why it matters

Because the header is set at server or CDN level, one misconfigured rule can silently noindex an entire path prefix, file type or subdomain rather than a single page — and routine page-level audits miss it unless response headers are inspected on every URL. Google honors X-Robots-Tag fully, including for non-HTML files (PDFs, images) where a meta tag cannot be placed. Common accidents: a staging-wide noindex header carried into production, an over-broad CDN edge rule, or an SEO/security plugin emitting the header for more URLs than intended.

How to fix it

1. Confirm the header and its value: curl -I https://example.com/page and look for X-Robots-Tag: noindex.
2. Trace where it is set — web-server config, CDN edge rule or Worker, CMS plugin, or app middleware.
3. Remove or scope the rule to the intended path only (e.g. Nginx add_header X-Robots-Tag "noindex, nofollow"; inside a location /admin/ block).
4. Re-crawl the affected URLs, confirm the header is gone, then request indexation in Search Console.

Example

HTTP/2 200
x-robots-tag: noindex
<!-- page source has NO noindex — the block is header-only -->

HTTP/2 200
# remove the X-Robots-Tag rule at the server/CDN if the page should rank

A header-level noindex doesn't appear in page source, so it's easy to miss. One broad server/CDN rule can deindex an entire path.

When it's not a problem

X-Robots-Tag: noindex is correct and expected for staging environments, admin/authenticated sections (/wp-admin/, /dashboard/), non-HTML assets that should not surface in search, and some faceted/filter URLs. Do not remove the header without first confirming the page is meant to be indexed.

How Crawlinx detects it

We read the X-Robots-Tag response header separately from the meta tag. A noindex set only in the header — invisible in page source — is reported here so header-level blocks aren't missed.

How common is it?

105 audited sites in our corpus currently show this issue. The breakdowns below show which platforms, gatekeepers, verticals and countries are most exposed.

Related guides

• Meta Robots & X-Robots-Tag: noindex/nofollow Guide