Rate-limited (429): your site throttled the crawler

What it means

HTTP 429 means the server sent a rate-limit response to the crawler, not that the page itself is broken. Google treats 429 as a server error rather than a normal 4xx, so crawlers slow down their visit rate.

Why it matters

Each 429 response reduces crawl capacity, meaning pages are fetched less often and indexed more slowly. If robots.txt itself returns 429, crawling of the entire site can halt for roughly 12 hours. The crawler falls back to a cached copy for up to 30 days, which delays fresh content discovery.

How to fix it

1. Check server logs for rate-limit headers and verify the crawler's IP resolves back to googlebot.com.
2. Ensure robots.txt returns 200, not 429 or 5xx — a 429 on robots.txt stops crawling the whole site.
3. Add retry logic or increase per-client rate limits for Googlebot IP ranges.
4. Serve a proper Retry-After header so the crawler knows when to resume.

Example

GET /category/shoes
429 Too Many Requests
Retry-After: 120
# a WAF or rate limiter throttled the crawler

# Allowlist verified crawlers and set a sane per-IP rate:
# nginx: limit_req_zone with a higher burst for known bot UAs/IP ranges
200 OK

A 429 is a server throttling response, not a page defect. Verify legitimate crawlers (Googlebot, CrawlinxBot) by reverse-DNS and raise their rate limit rather than blocking them.

When it's not a problem

A brief 429 during a traffic spike is usually harmless — the crawler will retry. If it only appears on deep pages or rarely accessed URLs, it may not affect your index at all.

How Crawlinx detects it

We flag any URL whose response status is HTTP 429 (Too Many Requests). This is a site-side throttle telling the crawler to slow down — it reflects your rate limiting, not a problem with the page content itself.

How common is it?

Not yet observed in our audit sample. We publish this explainer because the check runs on every crawl; once an audited site trips it, this page will break the issue down by platform, gatekeeper, vertical and country.

FAQ

Is a 429 bad for SEO?

If Googlebot hits repeated 429s it backs off crawling, so new and updated pages get discovered slower. Persistent 429s can shrink your effective crawl budget.

Why did the crawler get rate-limited?

A WAF, CDN or origin rate limiter treated the crawl as too aggressive. Bot-management rules, low per-IP request caps, or challenge pages are the usual triggers.

How do I stop throttling legitimate bots?

Verify bots by reverse-DNS (Googlebot, CrawlinxBot publish their ranges) and allowlist them with a higher rate. Keep tight limits for unverified traffic, not for confirmed search crawlers.