Meta's AI crawlers
Meta's AI crawlers: Meta-ExternalAgent, Meta-ExternalFetcher and Meta-WebIndexer
Meta's AI crawlers explained: what Meta-ExternalAgent, Meta-ExternalFetcher and Meta-WebIndexer do, how robots.txt applies, and which ones to allow.
AI search crawler
Meta-WebIndexer is a Meta crawler that navigates the web to improve Meta AI search result quality for users. Meta says allowing it in robots.txt helps Meta cite and link to your content in Meta AI's responses.
Meta instructs site owners to add a robots.txt disallow for the relevant crawler to block it, with no caveat for this one. Meta says allowing it helps Meta cite and link to your content in Meta AI.
User-agent: meta-webindexer Allow: /
Meta does not publish a way to verify Meta-WebIndexer. Treat the user agent alone as a claim, not proof.
Sill Agent Intelligence
Sill Agent Intelligence is building Meta-WebIndexer's behaviour profile from verified traffic across the Sill network: reach, identity checks, activity trend and the pages it targets. It appears here automatically when ready.
Meta's AI crawlers
Meta's AI crawlers explained: what Meta-ExternalAgent, Meta-ExternalFetcher and Meta-WebIndexer do, how robots.txt applies, and which ones to allow.
Only if you do not want Meta AI to cite and link to your pages. Meta says allowing it helps Meta cite and link to your content in Meta AI's responses.
Meta's crawler page lists its user agent strings but does not publish IP ranges or a verification method for it.