Access blocked (403 / WAF): crawler couldn't fetch

What it means

A WAF (web application firewall) is returning HTTP 403 to Googlebot and other crawlers, preventing them from fetching the page. This is not a defect in the page itself — the HTML, content, and metadata are fine — it is a network-level block that stops the crawler from seeing the page at all.

Why it matters

When crawlers cannot fetch a page, they cannot index it, which means the page will not appear in search results and will not be available for AI Overviews or AI Mode to cite. A persistent 403 also wastes crawl budget: Googlebot keeps retrying the same URLs without extracting any value from them.

How to fix it

1. Verify the WAF is returning 403 specifically to Googlebot and Google-InspectionTool (check UA header, source IP, and reverse-DNS against googlebot.com / google.com / googleusercontent.com).
2. Add an allow rule for Googlebot's user-agent and the Google-InspectionTool token so they are not blocked by the WAF.
3. Confirm the WAF is not confusing Googlebot's UA string with a generic bot — Google requires UA header, source IP, and reverse-DNS verification, not just the UA string alone.
4. Test with a curl request using Googlebot's UA and a known Googlebot IP to reproduce the 403, then re-test after the WAF rule change.

When it's not a problem

If the WAF is only blocking low-priority crawlers (GoogleOther, Storebot-Google) while Googlebot still gets 200, the page is indexed and the 403 is a false alarm. Also, if you intentionally use 403 to throttle non-essential bots or to serve a lightweight version to crawlers, this is acceptable — it does not harm indexing.

How common is it?

1 audited site in our corpus currently show this issue. The breakdowns below show which platforms, gatekeepers, verticals and countries are most exposed.