Internal links to robots.txt-disallowed URLs: fix the leak

What it means

One or more URLs match a Disallow rule in robots.txt yet appear as internal link targets in the crawl. Googlebot encounters the links, records the target URLs, checks robots.txt for each, and rejects them without fetching the body — each discovery is a crawl signal consumed with no indexing return.

Why it matters

Disallowed URLs are not invisible to Googlebot at the link level: it still records the URL and checks robots, and that check consumes crawl bandwidth. On a site whose faceted navigation disallows every parameter variant but links to them from product pages, this can generate thousands of useless discovery events per crawl pass. The equity cost is concrete too — Google cannot crawl the disallowed page to find its outlinks, so equity flowing in is absorbed and lost. Google's crawl-budget documentation names this among its listed crawl-efficiency problems.

How to fix it

1. Remove the internal links to disallowed URLs — especially navigation links to search, filter, or admin paths.
2. If the pages were disallowed by mistake and should be crawlable, remove or narrow the Disallow rule (and confirm they are not also noindex).
3. For soft exclusions (thank-you, print versions) prefer noindex over Disallow — Disallow prevents Google from reading the canonical or robots meta.
4. Re-crawl and verify inlink counts to disallowed targets drop for the affected URL patterns.

When it's not a problem

Some disallowed URLs legitimately appear in navigation — /login/ or /admin/ are commonly disallowed and linked from the header. A handful of such links on a site of any size is not a crawl-efficiency problem; the robots check is trivial next to a full fetch. It becomes material when a pattern of links (navigation, pagination, facets) generates dozens or hundreds of disallowed-URL discoveries per pass. Crawlinx reports the referring pages per target so incidental single links are easy to distinguish from systematic template issues.

How common is it?

2116 audited sites in our corpus currently show this issue. The breakdowns below show which platforms, gatekeepers, verticals and countries are most exposed.