How to control AI crawlers with robots.txt: GPTBot, Google-Extended and the retrieval tradeoff

AI crawlers fall into three distinct functions, and treating them as one bucket is the most common and most expensive mistake. Blocking a training crawler keeps you out of a model's training data. Blocking a retrieval crawler removes your eligibility to be cited in live AI answers — a far larger cost. This guide names the bots by function and gives real robots.txt examples for each decision.

The three-function model

Not every AI user agent does the same job. Before you write a single Disallow, decide which of these three you are affecting.

1. Search / retrieval bots — the ones you almost always want. These fetch pages to answer live queries and surface clickable citations. Examples: OAI-SearchBot (ChatGPT search), Claude-SearchBot, and PerplexityBot. Google's AI Overviews and AI Mode use ordinary Googlebot, so allowing Googlebot already covers Google's AI features. If you want to be cited in AI answers, these bots must be allowed.

2. Training-only crawlers — the ones you may block. These collect content to train future models and do not directly determine whether you appear in a live answer. Examples: GPTBot (OpenAI training), ClaudeBot (Anthropic training), CCBot (Common Crawl), Bytespider (ByteDance), and Meta-ExternalAgent. Blocking these is a defensible content-licensing decision. It does not remove you from live AI answers.

3. User-triggered fetchers — inconsistent, judge case by case. These fetch a page because a specific user asked an assistant to look at it. Examples: ChatGPT-User, Claude-User, Perplexity-User, and Google's new Google-Agent (March 2026), an agent that browses on a user's behalf using the identity agent.bot.goog. Vendor policies on whether these respect robots.txt are inconsistent, and their traffic is user-initiated rather than mass crawling.

Opt-out tokens are not crawlers

A critical distinction: Google-Extended and Applebot-Extended are not crawlers at all. They are opt-out tokens. Disallowing Google-Extended opts your content out of Gemini model training — it does not affect Search ranking, Googlebot crawling, or AI Overview eligibility. If your goal is to stay out of AI Overviews specifically, do not touch robots.txt; use the Search Console opt-out toggle (June 2026), which excludes content from AI features without harming organic rank.

Real robots.txt examples

For a site that wants AI visibility — the default recommendation for most publishers — allow the retrieval bots explicitly and, optionally, block the training-only crawlers:

# Allow AI search / retrieval bots (these drive citations)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Optionally block training-only crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

If instead you want to maximize reach and let everything in, the simplest correct policy is to add nothing AI-specific and make sure you are not accidentally blocking these agents with a broad User-agent: * Disallow.

The tradeoff, stated plainly

The reason to be careful is asymmetric cost. Blocking a training bot — GPTBot, ClaudeBot — costs you presence in a future training set, which is speculative. Blocking a retrieval bot — OAI-SearchBot, PerplexityBot — removes your eligibility to be cited in that platform's live answers today. Many sites block "all the AI bots" in one sweep, believing they are protecting their content, and quietly remove themselves from ChatGPT and Perplexity citations in the process.

Be equally honest about the upside: unblocking retrieval bots restores eligibility, not citations themselves. What actually drives citations is off-site authority — brand mentions are the strongest measured correlate — not the presence of any bot rule. Allowing retrieval bots is necessary but not sufficient. The GEO and AEO guide covers what earns citations once you are eligible.

Two enforcement caveats. First, robots.txt is a request, not a wall — some crawlers ignore it, and stealth crawling has been documented, so genuinely sensitive content needs WAF or server-side blocking, not a Disallow. Second, do not block CSS or JavaScript in the same file: Google's rendering service needs those resources, and blocking them degrades how your pages are understood. Crawlinx flags this with robots.blocks_css_js.

Where llms.txt fits — and does not

llms.txt is marketed as the way to talk to AI crawlers. The data does not support it as an SEO or GEO tactic. Roughly 97% of llms.txt files received zero requests in a month (Ahrefs, 137k domains, May 2026), and among the few that were requested, the top requester was SEO audit tools at 21.7% — not AI crawlers. Google does not consume it and has said it will not. The one legitimate use is developer-facing product documentation fetched by coding assistants such as Cursor, Copilot, and Claude Code. Treat its absence as documentation hygiene, not a ranking gap. Crawlinx reports it at notice severity through agent.llms_txt_missing for exactly that reason.

Looking ahead, Web Bot Auth (experimental, May 2026) is an IETF-draft cryptographic scheme for verifying bot identity. Because user-agent strings are trivially spoofed, cryptographic verification is the likelier long-term path for distinguishing legitimate agents from impostors — more consequential for bot policy than llms.txt.

How Crawlinx helps

Crawlinx checks the honest levers rather than the marketed ones.

• geo.ai_bots_blocked — flags when your robots.txt blocks retrieval bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot), which removes citation eligibility. It distinguishes these from training-only crawlers so you see the tradeoff before you commit to it.
• robots.blocks_css_js — catches CSS or JavaScript blocked in robots.txt, which breaks rendering for Google and AI retrieval alike.
• agent.llms_txt_missing — reports llms.txt presence as informational for developer-doc use cases, never as a ranking or citation requirement.

Takeaway

Sort AI crawlers into retrieval, training, and user-triggered before writing rules. Allow retrieval bots if you want citations; block training bots only as a deliberate content-licensing choice; and remember Google-Extended is a training opt-out token, not a crawler. Do not block CSS or JavaScript, and do not expect llms.txt to move anything. For what actually earns citations once you are eligible, read the GEO and AEO guide; for the broader 2026 context, the Google Search updates tracker.