Training Opt-Out Is Not Search Opt-Out: A robots.txt Playbook for AI Crawlers

Robots.txt used to be a crawl-budget file. Now it works as a licensing document. Every major AI vendor has split its bots by job (training, search indexing, user-triggered fetching), and each job gets its own token. A robots.txt written for 2019 makes 2026 decisions by accident.

The common failure isn’t ignorance. It’s ambiguity. Someone pastes a “block AI bots” snippet, training opt-out and search visibility get bundled into a single gesture, and the site quietly disappears from the answer engines its buyers use. Or the reverse: a site opens everything to stay visible and hands over training rights it meant to keep.

This guide gives you the mechanics that most snippets get wrong, a decision table organized by job rather than by vendor, three ready-to-adapt postures, and the verification steps that tell you whether your file actually does what you think.

The three jobs behind the tokens

Sort every token by what it does, not by whose it is:

Sources: Google common crawlers, OpenAI crawlers, Anthropic crawler help, Perplexity crawlers. Verified November 2026. Re-verify on the day you change your file.

A few details in those docs change real decisions:

• OpenAI treats its settings as independent: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot for training. Sites opted out of OAI-SearchBot won’t be shown in ChatGPT search answers, though they can still appear as navigational links.
• Perplexity says PerplexityBot exists to surface and link sites in its search results and is not used to crawl content for AI foundation models.
• Anthropic says restricting ClaudeBot signals that future materials should be excluded from training, while disabling Claude-SearchBot “may reduce your site’s visibility and accuracy in user search results.”
• User-triggered fetchers behave differently. OpenAI notes that robots.txt rules may not apply to ChatGPT-User, because a person initiated the action, and Perplexity says Perplexity-User generally ignores robots.txt. Don’t build a strategy that depends on blocking them through robots.txt alone.

The Google detail most snippets get wrong

Google-Extended is not a crawler. Google describes it as a standalone product token used “in a control capacity”. It has no user-agent string of its own, and it governs whether crawled content may be used to train future Gemini models and to ground Gemini Apps and Vertex AI. Google is explicit that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” (source).

The corollary matters more: you cannot opt out of AI Overviews or AI Mode with Google-Extended. Google says AI is integral to Search, so robots.txt directives for Googlebot are the control for Search crawling, and the controls that limit what’s shown are nosnippet, data-nosnippet, max-snippet, and noindex (Google: AI features and your website). If a stakeholder asks you to “keep us out of AI Overviews but stay in Search,” the honest technical answer is snippet controls, page by page. Robots.txt can’t do it.

One more forward-looking note from the same Google page: Google intends to support the IETF AIPREF training-preference signal (user-agent: __usage_ai-training) in early 2027. Anthropic says that, as of October 30, 2026, it honors a machine-readable “no AI training” preference, while still giving precedence to rules that name ClaudeBot directly. Training preferences are moving toward a shared vocabulary. Plan for your file to carry both vendor tokens and the shared signal for a while.

The parsing mechanics behind silent failures

RFC 9309, the robots exclusion standard, sets three rules that cause most of the bugs:

1. A crawler obeys the group that matches its token. If there’s none, it falls back to *. So when your file has User-agent: * / Disallow: / plus a carve-out for GPTBot, every bot you didn’t name, including the search bots you wanted, inherits the blanket block.
2. Multiple groups for the same token are merged. A second User-agent: OAI-SearchBot group added later by a plugin doesn’t replace the first. Its rules combine with it.
3. The most specific path match wins; on a tie, allow wins. A broad Disallow: / with Allow: /blog/ keeps the blog open for that group.

The decision rule that follows: name every token you care about explicitly. Treat * as the policy for strangers, never for the bots your visibility depends on.

Three postures, ready to adapt

Posture A: Visible, not trained. Stay in every answer engine, and decline training wherever a separate control exists.

User-agent: Googlebot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /

Posture B: Fully open. Allow everything, and state it explicitly so a future plugin’s * rule can’t override your intent.

Posture C: Google Search only. Allow Googlebot, disallow the other search and training tokens. This is a legitimate choice for some publishers. Just make it on purpose, knowing it removes you from those engines’ answers.

Most businesses that sell expertise belong in Posture A. They want to be found, they want their answers attributed, and they’d rather not donate their content to the next foundation model.

The layer robots.txt can’t see

Your robots.txt can say “allow” while your CDN says “403.” Google’s AI-features guidance lists “ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure” as a fundamental. Bot-management features, WAF rules, and rate limits routinely challenge the very search bots you allowed. Perplexity’s crawler docs include WAF configuration steps for exactly this reason.

So verify in three passes:

1. Logs: search access logs for each token and check the status codes. A 200 means the bot read the page. A 403, a 429, or a challenge page means your infrastructure answered before robots.txt mattered.
2. IP verification: user-agent strings can be spoofed. OpenAI and Perplexity publish IP ranges for their bots, and Google publishes ranges for its common crawlers. Match the claimed bot against the published list before you trust what the logs say.
3. Extractability: fetch your key pages without JavaScript. If your services, location, and differentiator only exist after client-side rendering, an allowed bot can still come away with an empty shell.

Edge cases that deserve a written decision

Three situations trip up teams that otherwise get the main file right:

• Subdomains. Robots.txt is scoped per host. Anthropic says explicitly that opting out requires modifying robots.txt on each subdomain, and the same logic applies to every crawler. Your docs portal, help center, and shop subdomain each need their own file, and their own row in the decision table.
• Staging and preview hosts. A staging site that’s fully open to search bots can leak unfinished pages into answer engines. Disallow everything there, and back it with authentication, because robots.txt is a request, not a lock.
• Crawl rate versus access. If the concern is load rather than licensing, throttle instead of blocking. Anthropic supports the non-standard Crawl-delay directive, so you can slow ClaudeBot without removing your content from search. Blocking a search bot to fix a server-load problem trades a solvable infrastructure issue for lost visibility.

Each of these is a one-line decision. Write it down anyway: the next person to edit the file will thank you, and a written decision is much harder for a plugin to undo silently.

The decision table you’ll actually maintain

Keep one table, dated, next to your robots.txt in version control:

The table does two jobs. It forces a written reason for every choice, which ends the “who added this line?” archaeology. And it gives you a re-verification schedule: OpenAI and Perplexity both say changes can take about 24 hours to reflect, and token definitions do change. Review quarterly, and whenever a vendor announces a new bot.

Worked example: a consultancy blocks itself

Illustrative example. A boutique consultancy installs a security plugin that writes User-agent: * / Disallow: / for “AI protection,” then adds a Googlebot allow. Search traffic looks fine. Three months later, the team notices that ChatGPT and Perplexity answers recommend two competitors and never them.

The diagnosis takes ten minutes with RFC 9309 in hand: OAI-SearchBot, Claude-SearchBot, and PerplexityBot have no named groups, so they fall back to * and are blocked. The fix is Posture A, with each search token named explicitly. The log check then shows a CDN bot-fight rule still challenging PerplexityBot, so a second fix lands at the WAF. Two layers, two fixes, and a table that records why both exist.

Score access alongside everything else

Crawler access is one of eight areas that decide whether answer engines can understand and cite you. Organa Obscura’s AI-Visibility (AEO) Audit puts it in context. Its 27-page guide includes an AI crawler token table that covers what Google-Extended does and doesn’t control, and a 60-check Sheets/Excel workbook scores access next to entity clarity, structured data, answer-ready content, and authority signals. Its Fix Plan ranks the work automatically, so a robots.txt fix doesn’t jump the queue ahead of a bigger entity problem, or fall behind a smaller one. The free one-page Quick Check is included for fast re-checks.

If you also want the prompts to produce the answer-ready content these bots will retrieve, the SEO Pro Kit bundles the Audit with the SEO & Content Prompt Pack’s 120 prompts. Browse everything at organa.agency/products/.