Search engine crawler

Googlebot

Googlebot is Google's main crawler. It reads pages to build the Google Search index, which also feeds products such as Images, Video, News and Discover.

Sources: Google's own documentation and the Sill agent registry · checked 15 Nov 2026

Company
Google
Acts for
Google Search
Follows robots.txt
Yes
How to verify
Published IP list, Reverse DNS

robots.txt

Google says its crawlers always obey robots.txt rules when crawling automatically.

User-agent: Googlebot
Allow: /

How to check it is really Googlebot

How verification works →

Sill Agent Intelligence

What it does on websites

Live field data · through 8 Nov 2026 · refreshed weekly

Reach
Across the Sill network
First observed
October 2026
Status
Active now

Identity check: traffic presenting as Googlebot

  • Verified 35%
  • Failed verification 20%
  • Unverified 45%

Where it goes

  • Home page20%
  • Other pages80%

Source: Sill Agent Intelligence, built from verified agent traffic across the Sill network. Figures are aggregated and anonymised; no website is ever identified.

Other Google agents

Read more on Signals

Questions about Googlebot

Should I block Googlebot?

Almost never, unless you want your pages out of Google Search. Blocking it removes you from the index that powers Search.

How do I check a request is really Googlebot?

Google says to check the IP against its published ranges and confirm the reverse DNS hostname ends in googlebot.com or geo.googlebot.com.

Sources

  1. Google: Common crawlers · checked 15 Nov 2026

Official documentation: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers · Spotted an error? Tell us

See which AI agents visit your site, and decide what they may do.

Scan your site free