How to Control AI Crawlers with robots.txt (Without Losing Your Citations)

llms.txt invites. robots.txt restricts. If you want to actually control what AI systems do with your content, robots.txt is still the only mechanism the major crawlers claim to honour — and the only one they have publicly committed to.

The catch is that “block AI” is no longer one decision. Every major vendor now runs several crawlers with different jobs, under different user-agent tokens. Treating them as one bucket is how sites accidentally delete themselves from AI search results while trying to opt out of model training.

The three jobs a crawler can have

Before the token list, the taxonomy. Vendors split crawling three ways:

• Training — bulk collection to train or fine-tune a model. Blocking this costs you nothing in traffic.
• Search / indexing — building the index that produces cited answers. Blocking this costs you visibility.
• User fetch — one page, fetched live because a person asked about it. Blocking this breaks the experience for someone who is already trying to read you.

Most sites want to make different choices for each. The token split is what makes that possible.

The token list, by vendor

Two entries deserve a footnote. Google does not let you separate classic search from AI Overviews:Googlebot feeds both, andGoogle-Extended only controls Gemini and Vertex AI training. There is no way to appear in Google Search while opting out of AI Overviews.

And Applebot-Extended has a trap: when it has no rules of its own, it inherits Applebot's. If you wrote a permissive Applebot block and assumed Apple was not training on you, check again.

Three policies you can copy

1. Stay visible, opt out of training

The most common choice for publishers and marketing sites: be citable, do not be training data.

# Training — blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

# Search and live fetches — allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /

2. Maximum AI presence

For docs, developer tools, and anyone whose goal is to be the answer: allow everything from the major vendors, block only the bulk scrapers that give nothing back.

User-agent: *
Allow: /

User-agent: CCBot
User-agent: Bytespider
Disallow: /

3. Protect one section only

Paywalled or member content out, everything else in. Note that a path-scoped Disallow is not a block — it leaves the rest of the site open, which is usually the point.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /members/
Disallow: /premium/

User-agent: *
Allow: /

What robots.txt cannot say

Allow and Disallow govern fetching. They cannot express what a crawler may do with the bytes afterwards — and “you may read this to answer a question, but not to train on it” is exactly the distinction most publishers want.

Two efforts fill that gap: Cloudflare's Content Signals, already deployed across millions of domains, and the IETF's AIPREF working group, whose Content-Usage rule is on the standards track. Both are covered here.

One thing worth being honest about: all of this is advisory. robots.txt has never been enforceable. The major vendors state that their training crawlers respect it, and the evidence is that they largely do — but if you need a guarantee rather than a norm, that is a server-side decision (403 by user-agent, or a WAF rule), not a robots.txt one.

Check your file

Our validator now reads robots.txt and builds this table for you: every vendor, every token, and whether your file allows it, blocks it, covers it by wildcard, or never mentions it. It also flags the search-blocked-with-training mistake by name, and checks any Content-Signal or Content-Usage lines you have added.