Content-Signal
Content-Signal in robots.txt: ai-train, ai-input and search explained
How the Content-Signal line in robots.txt works: ai-train, ai-input and search, Cloudflare's defaults, the IETF aipref drafts, and who honors them.
robots.txt control token
Google-Extended is a robots.txt token, not a crawler. Publishers use it to control whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps.
Google says crawling is done with existing Google user agents; Google-Extended only tells Google how crawled content may be used.
User-agent: Google-Extended Allow: /
Google-Extended is only a rule name in robots.txt. It never visits, so a request that claims to be Google-Extended is not genuine.
Google-Extended is a robots.txt rule, not a visitor, so there is no website activity to show.
Content-Signal
How the Content-Signal line in robots.txt works: ai-train, ai-input and search, Cloudflare's defaults, the IETF aipref drafts, and who honors them.
Block AI agents
Should you block AI bots and agents? A decision guide for store owners: training crawlers, AI search bots and shopping agents each need a different answer.
reverse DNS
Verify AI bots by IP range and reverse DNS: the official IP lists and DNS checks for OpenAI, Anthropic, Google, Perplexity, Bing, Apple, Amazon and more.
No. Google says it does not affect inclusion in Google Search and is not a ranking signal.
No. It has no user agent of its own, so a request that claims to be Google-Extended has failed verification.