Blocked by robots.txt: how to find and fix it

What "blocked by robots.txt" means and how to catch pages you accidentally hid from Google. We flag every robots-blocked URL for free.

1765

audited sites affected

What it means

A robots.txt disallow rule is preventing Googlebot from fetching this page's content. It does not mean the page is hidden from search results — a disallowed URL can still be indexed if it appears in external links.

Why it matters

When Googlebot cannot crawl the page, it cannot read meta directives like noindex, nor can it extract content for indexing or AI-agent access. Over time, persistent crawl blocks reduce how often the page is revisited, which slows updates and freshness signals.

How to fix it

Open your robots.txt and locate the Disallow: pattern matching this URL.
If the page should be crawled, remove or narrow the disallow rule (e.g., Disallow: /blog/ → Allow: /blog/post-name/).
If the page should stay out of results, add noindex in the HTML <head> or as an X-Robots-Tag header — not in robots.txt.
Verify the fix with the URL Inspection tool and request a re-crawl.

Example

Before — robots.txt

User-agent: *
Disallow: /blog/

After — robots.txt

User-agent: *
Disallow: /cart/
Disallow: /search
# /blog/ removed so Googlebot can crawl it

Disallow blocks crawling, not indexing — a blocked URL can still appear in results without a snippet. Only block what you truly never want crawled.

When it's not a problem

It is fine to leave this flagged when the page is genuinely not useful for search (e.g., a staging URL, a parameterized variant, or a page whose content is fully represented in another URL). Also fine if the page is already indexed and you simply want to reduce crawl traffic to it.

How Crawlinx detects it

We evaluate the site's robots.txt rules against each URL. When a URL matches a Disallow rule for our user-agent, we flag it as blocked from crawling.

How common is it?

1765 audited sites in our corpus currently show this issue. The breakdowns below show which platforms, gatekeepers, verticals and countries are most exposed.

FAQ

Does robots.txt disallow remove a page from Google?: No. It stops crawling, but a disallowed URL can still be indexed (URL only, no snippet) if it's linked. To remove a page, use noindex and let it be crawled — don't block it.

Related guides

Breakdowns

CMS

View breakdown →

Audit your own site — free

157 checks, internal PageRank, render-diff. No signup, results in ~30s.

Scan your site →