Robots.txt blocks AI search crawlers: fix AI citations

What it means

Your robots.txt disallows an AI *search* crawler — such as OAI-SearchBot (ChatGPT search), PerplexityBot, or Claude-SearchBot. These are the bots that fetch pages to cite in AI-generated answers. Blocking them removes your site from the pool of sources those systems can quote, so you lose visibility in AI answers even when you rank well in classic search.

Why it matters

AI answer engines split their crawling into two roles, and conflating them is the most common mistake. Training crawlers (GPTBot, ClaudeBot, CCBot) collect data to train models; blocking them is a legitimate privacy or licensing choice with no direct effect on live citations. Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) fetch pages at answer time to retrieve and cite current information — allow these if you want citations. When a search bot is disallowed the engine cannot retrieve your page to quote it, and your content is passed over for a competitor's. As AI Overviews, ChatGPT search, and Perplexity take a growing share of informational queries, being uncitable is a genuine visibility loss. Note the honest limit: allowing these bots is necessary for AI citation but does not guarantee it — citation depends on many factors beyond crawl access.

How to fix it

1. Open robots.txt and find any Disallow rules under user-agent groups for the AI search bots.
2. Remove the disallow, or add an explicit Allow: /, for the search crawlers you want citing you (OAI-SearchBot, PerplexityBot, Claude-SearchBot).
3. Keep training-bot blocks if that is your policy — the two decisions are independent.
4. Re-fetch robots.txt and confirm the search-bot groups no longer disallow /.

When it's not a problem

Crawlinx fires this only for the search bot set, never for training bots — blocking GPTBot or ClaudeBot is a defensible decision and is not flagged. If you have deliberately opted out of AI search visibility, this notice is expected and you can dismiss it; it exists to catch the far more common case of blocking search bots by accident, for instance a blanket disallow copied from a template.

How common is it?

2 audited sites in our corpus currently show this issue. The breakdowns below show which platforms, gatekeepers, verticals and countries are most exposed.

Related guides

• How to Control AI Crawlers: GPTBot, Google-Extended and robots.txt (2026)
• Robots.txt Guide: How Robots.txt Works