AI training crawler

GPTBot

GPTBot is OpenAI's web crawler. OpenAI says it collects content that may be used to train its generative AI foundation models. It visits sites automatically, not in response to a person's request.

Sources: OpenAI's own documentation and the Sill agent registry · checked 15 Nov 2026

Company
OpenAI
Acts for
OpenAI, for model training
Follows robots.txt
Yes
How to verify
Published IP list

robots.txt

OpenAI says disallowing GPTBot in robots.txt indicates a site's content should not be used to train its models.

User-agent: GPTBot
Allow: /

How to check it is really GPTBot

How verification works →

Sill Agent Intelligence

What it does on websites PROFILE IN PROGRESS

Sill Agent Intelligence is building GPTBot's behaviour profile from verified traffic across the Sill network: reach, identity checks, activity trend and the pages it targets. It appears here automatically when ready.

Other OpenAI agents

Read more on Signals

Questions about GPTBot

Should I block GPTBot?

Block it if you do not want your content used to train OpenAI models. Blocking GPTBot does not affect ChatGPT search visibility, which is controlled by OAI-SearchBot.

How do I know a request is really GPTBot?

Check the visitor's IP address against OpenAI's published list at openai.com/gptbot.json. A request that claims to be GPTBot but is not on that list has failed verification.

Sources

  1. OpenAI: Overview of OpenAI crawlers · checked 15 Nov 2026

Official documentation: https://developers.openai.com/api/docs/bots · Spotted an error? Tell us

See which AI agents visit your site, and decide what they may do.

Scan your site free