AI training crawler

ClaudeBot

ClaudeBot is Anthropic's crawler. Anthropic says it collects web content that could help train its generative AI models.

Sources: Anthropic's own documentation and the Sill agent registry · checked 15 Nov 2026

Company
Anthropic
Acts for
Anthropic, for model training
Follows robots.txt
Yes
How to verify
Published IP list

robots.txt

Anthropic says its bots honor industry-standard robots.txt directives and also supports the non-standard Crawl-delay.

User-agent: ClaudeBot
Allow: /

How to check it is really ClaudeBot

How verification works →

Sill Agent Intelligence

What it does on websites PROFILE IN PROGRESS

Sill Agent Intelligence is building ClaudeBot's behaviour profile from verified traffic across the Sill network: reach, identity checks, activity trend and the pages it targets. It appears here automatically when ready.

Other Anthropic agents

Read more on Signals

Questions about ClaudeBot

Should I block ClaudeBot?

Block it if you do not want future content used to train Anthropic models. Anthropic says this signals that your future content should be excluded from training datasets.

Can I block ClaudeBot by IP address?

Anthropic warns this may not reliably opt you out, because it stops the bot from reading your robots.txt. Use a robots.txt rule instead.

Sources

  1. Anthropic: Does Anthropic crawl data from the web? · checked 15 Nov 2026

Official documentation: https://support.claude.com/en/articles/8896518 · Spotted an error? Tell us

See which AI agents visit your site, and decide what they may do.

Scan your site free