Glossary

What are content signals (Content-Signal in robots.txt)?

Definition

Content signals are a line in robots.txt that states how a site owner wants its content used once it has been accessed, using three signals: search, ai-input and ai-train, each set to yes or no. Cloudflare introduced the Content Signals Policy in September 2025. The signals express preferences and do not technically block anything.

Why it matters

Classic robots.txt rules only say whether a bot may fetch a page. They don't say what may happen to the page afterwards. Content signals fill that gap. You can say, for example, that search engines may index your pages but your content should not be used to train AI models.

This matters because one crawler can serve several purposes. A content signal lets you state your wishes per use, not just per bot.

How it works

Cloudflare announced its Content Signals Policy on September 24, 2025, as a robots.txt addition that "allows you to express your preferences for how your content can be used after it has been accessed." It released the policy under a CC0 license, so anyone can use it. There are three signals:

  • search: "building a search index and providing search results". Cloudflare notes this "does not include providing AI-generated search summaries."
  • ai-input: "inputting content into one or more AI models".
  • ai-train: "training or fine-tuning AI models."

Each is set to yes or no. If you leave one out, Cloudflare says the website operator "neither grants nor restricts permission via content signal." For customers using its managed robots.txt, Cloudflare said it would serve search=yes, ai-train=no and leave ai-input unset.

Cloudflare is direct about the limits: "content signals express preferences; they are not technical countermeasures against scraping." It recommends combining them "with WAF rules and Bot Management."

Separately, the IETF's aipref working group is drafting a shared vocabulary for AI usage preferences, with categories such as AI training and search. At the time of writing that is a working draft, draft-ietf-aipref-vocab-08, and its labels differ from Cloudflare's. Google and Anthropic have pointed to a separate training signal based on the aipref work, not to Content-Signal: Google says it intends to support it with the robots.txt token __usage_ai-training in early 2027, and Anthropic says it began recognizing a no-AI-training signal on October 30, 2026.

Example

A store wants to appear in search and AI answers but keep its product copy out of model training. Its robots.txt includes:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Bots that honor the policy can index pages and quote them in answers, and are asked not to train on them. Like a robots.txt disallow, the signal is only a request. To enforce a block on a training crawler, the store also needs a firewall rule for it. See robots.txt for AI agents and should you block AI agents?

Also called: Content-Signal, Content Signals Policy, AI preferences in robots.txt

Sources

  1. Cloudflare blog: Giving users choice with Cloudflare’s new Content Signals Policy · checked 17 Nov 2026
  2. IETF: draft-ietf-aipref-vocab-08, A Vocabulary For Expressing AI Usage Preferences · checked 17 Nov 2026
  3. Google: Google's common crawlers · checked 17 Nov 2026
  4. Google Search Central blog: Supporting a shared AI training control for websites (October 30, 2026) · checked 17 Nov 2026
  5. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? · checked 17 Nov 2026

The Sill Signals Team · Updated 17 Nov 2026

See which AI agents visit your site, and decide what they may do.

Scan your site free