Content Signals and IETF AIPREF: Saying What AI May Do With Your Content

robots.txt answers one question: may you fetch this URL? It has no vocabulary for the question publishers actually care about now — you may read this to answer someone's question, but not to train a model on it.

Two efforts are filling that gap from opposite directions. Cloudflare shipped one unilaterally and put it on millions of domains overnight. The IETF is standardising the other properly, and slowly. They are compatible, they mean roughly the same things, and right now you should probably deploy the first while watching the second.

Cloudflare Content Signals

A Content-Signal: line in your robots.txt, with a comma-separated list of preferences:

Content-Signal: search=yes, ai-input=yes, ai-train=no

Three signals, each yes or no:

• search — building a search index, returning links and short excerpts. Explicitly not AI-generated summaries.
• ai-input — real-time use in a generated answer. This is retrieval-augmented generation: your page is read at question time and informs the reply.
• ai-train — training or fine-tuning a model.

That middle one is the interesting addition. Until Content Signals there was no way to say “cite me live, do not absorb me” — and for most publishers that is precisely the position.

IETF AIPREF

The IETF chartered an AI Preferences working group to do this properly: one vocabulary for expressing preferences, one set of mechanisms for attaching it to content, and defined rules for reconciling preferences that disagree.

Two drafts matter:

• draft-ietf-aipref-vocab — the vocabulary. Defines categories including train-ai and search, with y / n values.
• draft-ietf-aipref-attach — how to attach it: a Content-Usage HTTP response header, and a matching robots.txt rule.

In robots.txt it sits inside a user-agent group:

User-agent: *
Content-Usage: train-ai=n
Allow: /

Or as a header, which has a real advantage: it can be set per-resource, so a single page or a single PDF can carry a different preference from the rest of the site.

Content-Usage: train-ai=n

Both drafts are adopted working-group documents on the Proposed Standard track, with an August 2026 milestone to send the attachment spec to the IESG. That is real progress — but “on the standards track” is not “published”, and crawler support today is close to nonexistent.

Which should you deploy?

Add Content Signals now. It is one line, it costs nothing, a large share of the web already carries it, and it states an intent that Allow and Disallow cannot.

Wait on Content-Usage unless you enjoy being early. Broadcasting a preference no crawler reads adds a line that has to be kept consistent with everything else in the file. Revisit when the RFC publishes.

Keep Allow and Disallow as the load-bearing part. Both of these say what should happen to content a crawler is already permitted to fetch. Neither stops a fetch. If you want something not read, disallow it.

The failure mode to avoid

Declaring ai-train=no while leaving GPTBot and Google-Extended on Allow: /.

The signal is advisory; the Disallow rule is what crawlers act on. A file that says one thing in a signal and the opposite in its rules is not a policy — it is a contradiction, and the rules win. Our robots.txt validator flags exactly this pairing, alongside unknown signal keys and invalid values.