AI agents
Is your website ready for AI agents? A 2026 checklist
Is your website ready for AI agents? A 2026 checklist for store and business owners: access, readable pages, agent files, verification and order rules.
AI training crawler
GPTBot is OpenAI's web crawler. OpenAI says it collects content that may be used to train its generative AI foundation models. It visits sites automatically, not in response to a person's request.
OpenAI says disallowing GPTBot in robots.txt indicates a site's content should not be used to train its models.
User-agent: GPTBot Allow: /
Sill Agent Intelligence
Sill Agent Intelligence is building GPTBot's behaviour profile from verified traffic across the Sill network: reach, identity checks, activity trend and the pages it targets. It appears here automatically when ready.
AI agents
Is your website ready for AI agents? A 2026 checklist for store and business owners: access, readable pages, agent files, verification and order rules.
Cloudflare
How to allow AI agents like ChatGPT agent through Cloudflare: find what is blocked, allow verified agents, add skip rules, and keep bad bots out.
Content-Signal
How the Content-Signal line in robots.txt works: ai-train, ai-input and search, Cloudflare's defaults, the IETF aipref drafts, and who honors them.
Block it if you do not want your content used to train OpenAI models. Blocking GPTBot does not affect ChatGPT search visibility, which is controlled by OAI-SearchBot.
Check the visitor's IP address against OpenAI's published list at openai.com/gptbot.json. A request that claims to be GPTBot but is not on that list has failed verification.