Trust & security

Fake GPTBot, ClaudeBot and Googlebot: how user-agent spoofing works and how to catch it

Listen · 10 min

Narrated by an AI voice. Code examples are on the page.

Short answer

Any program can put GPTBot, ClaudeBot or Googlebot in its user-agent string, so the name alone proves nothing. To catch requests that claim a bot name but fail verification, check the IP address against the vendor's own published list (OpenAI, Anthropic, Perplexity, Google) or run a reverse and forward DNS lookup (Google, Bing). If the check fails, treat the request as an ordinary unverified visitor.

Key takeaways

  • A bot name in the user-agent is a claim anyone can send.
  • OpenAI, Anthropic, Perplexity and Google publish IP lists for their bots.
  • Google and Bing document reverse DNS checks with a forward confirmation.

How does user-agent spoofing work?

User-agent spoofing means sending another program's name in the User-Agent header. It takes no skill. The header is plain text that the sender chooses, so any script can claim to be any bot.

Google says it directly on its crawler documentation: "The HTTP user agent string can be spoofed." Cloudflare makes the same point, saying user-agent headers "are easily spoofed and are therefore insufficient for reliable identification."

Here is the user-agent OpenAI documents for GPTBot:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Anyone can copy that line into a script. Fake GPTBot requests look identical to real ones in your logs, unless you check something the sender can't easily change: the IP address it comes from.

Why would a request pretend to be a well-known bot?

Because well-known bots often get treated better. Many sites allow Googlebot, Bingbot or AI crawlers through firewalls, rate limits or paywalls so they can be indexed or cited. A request that borrows one of those names may get the same pass.

The people behind such requests vary. It might be a scraper, an SEO tool, a monitoring service or a developer testing a script. You usually can't tell which, and you don't need to. Google frames its own verification guidance around the risk that other parties may claim to be Google.

So the useful label is neutral: a request that claims to be GPTBot (or ClaudeBot, or Googlebot) but fails verification. It is not the named company's traffic, and you shouldn't treat it as if it were.

We have not found a vendor that publishes a figure for how much bot traffic is spoofed, so we don't give one.

How do you verify GPTBot, ClaudeBot and PerplexityBot?

Match the visitor's IP address against the JSON list the vendor publishes for that exact bot. If the user-agent claims a bot and the IP is inside that bot's ranges, the request is very likely genuine. If not, it fails verification.

Claimed bot Vendor Published IP list
GPTBot OpenAI https://openai.com/gptbot.json
OAI-SearchBot OpenAI https://openai.com/searchbot.json
ChatGPT-User OpenAI https://openai.com/chatgpt-user.json
ClaudeBot, Claude-User, Claude-SearchBot Anthropic https://claude.com/crawling/bots.json
PerplexityBot Perplexity https://www.perplexity.ai/perplexitybot.json
Perplexity-User Perplexity https://www.perplexity.ai/perplexity-user.json

A few vendor-specific notes, from each vendor's own docs:

  • OpenAI publishes one list per bot. Check against the list for the bot the request claims to be, not any OpenAI list. More detail on GPTBot, OAI-SearchBot and ChatGPT-User.
  • Anthropic publishes one shared list. It says that if a crawler's source IP is on the list, that "indicates that the crawler is coming from Anthropic". It also warns that blocking these IPs "may not work correctly or persistently guarantee an opt-out", because it stops its bots from reading your robots.txt. See ClaudeBot.
  • Perplexity recommends you "combine both User-Agent string matching and IP address verification". See PerplexityBot.

In practice:

  1. Download the list for the claimed bot. Each file contains IP ranges in CIDR form, such as 192.0.2.0/24.
  2. Check whether the visitor's IP falls inside any range in that file.
  3. Refresh the lists on a schedule, daily is a sensible default, because vendors add and remove ranges.

For ChatGPT specifically, our guide on how to tell if a visitor is really ChatGPT also covers signed ChatGPT agent requests.

How do you verify Googlebot?

Google documents two methods: a reverse and forward DNS lookup, or a match against its published IP ranges. Either one catches requests that claim to be Googlebot but fail verification.

Reverse DNS, step by step (Google's method, using the host command):

  1. Run a reverse lookup on the IP address from your logs.
  2. Check the hostname ends in googlebot.com, google.com or googleusercontent.com.
  3. Run a forward lookup on that hostname.
  4. Confirm it resolves back to the original IP address.
host <ip-from-your-logs>
host <hostname-from-step-1>

Step 4 matters. Anyone who controls an IP block can set its reverse DNS to say almost anything. Only the forward lookup confirms it, because only Google controls what googlebot.com and google.com names resolve to.

Be more precise for Googlebot itself. Google says its common crawlers use hostnames like crawl-*-*-*-*.googlebot.com or geo-crawl-*-*-*-*.geo.googlebot.com. Hostnames under gae.googleusercontent.com belong to user-triggered fetchers, and Google's own example is a site on Google Cloud fetching feeds at a user's request. So a request that says "Googlebot" but resolves to googleusercontent.com is not Googlebot.

IP lists. Google publishes separate JSON files for each group, including:

  • Common crawlers, which includes Googlebot: https://developers.google.com/static/crawling/ipranges/common-crawlers.json
  • Special-case crawlers: special-crawlers.json
  • User-triggered fetchers: user-triggered-fetchers.json and user-triggered-fetchers-google.json
  • User-triggered agents, which covers Google-Agent: user-triggered-agents.json

All sit under the same ipranges path. Our explainer on Google-Agent covers the agent list in more detail.

How do you verify Bingbot?

Use reverse DNS, then confirm with a forward lookup. Bing's verification guide says a genuine Bing crawler's IP resolves to a hostname ending in search.msn.com, and that hostname must resolve back to the same IP.

  1. Run a reverse lookup on the IP (Bing's guide shows nslookup on Windows and host on Linux).
  2. Check the hostname ends in search.msn.com.
  3. Run a forward lookup on that hostname.
  4. Confirm it returns the original IP.

Bing's guide also advises against hard-coding a fixed list of addresses, because the ranges Bing uses can change at any time. Bing also publishes its current ranges at https://www.bing.com/toolbox/bingbot.json. Bing offers a Verify Bingbot lookup page at https://www.bing.com/toolbox/verify-bingbot for one-off checks. See Bingbot.

What should you do with requests that fail verification?

Treat them as ordinary, unverified visitors. Don't call them out as a named company, and don't give them the access the real bot gets.

  • Never allow by name alone. If a firewall rule, rate-limit exception or paywall bypass matches on a bot's user-agent, add an IP or DNS check to it.
  • Label, don't accuse. In logs and reports, mark these as "claims GPTBot, failed verification" rather than as GPTBot traffic. Your analytics stay honest.
  • Apply your normal rules. Rate limits, challenges and blocks that apply to unknown traffic apply here too.
  • Keep robots.txt as your opt-out tool. Vendors such as Anthropic say their crawlers follow it, while a request that fails verification may not. Our robots.txt guide covers which names to use.
  • Expect some agents with no bot name. Some AI agents drive an ordinary browser and send a normal Chrome user-agent. Our AI agent identity guide explains how signed requests help there.

For background on the terms, see user agent and verified bot. The AI agent directory lists the verification method for each bot.

How Sill helps

Sill shows you which AI agents and bots visit your site and the pages they read. It verifies signed agent requests (Web Bot Auth) on Cloudflare and WordPress installs. To see how your site looks to AI agents now, run the free readiness scan, no account needed.

Questions people ask

Is it illegal to fake a Googlebot user agent?

We can't give legal advice, and the answer depends on where you are and what the requests do. For your site, the practical answer is the same either way: don't grant special access to any request that claims a bot name but fails verification.

Should I block a request that claims to be GPTBot but fails verification?

You don't have to block it outright. Treat it like any other unverified visitor and apply your normal rate limits and bot rules. The key is to never give it the access you give the real bot.

Does blocking ClaudeBot's IP addresses opt my site out of Anthropic's crawling?

Anthropic says blocking its IP addresses may not reliably work as an opt-out, because it stops its bots from reading your robots.txt. Anthropic recommends using robots.txt to opt out instead.

Can a googleusercontent.com hostname prove a request is Googlebot?

No. Google says Googlebot's hostnames end in googlebot.com. Hostnames under gae.googleusercontent.com belong to user-triggered fetchers, which Google says can include a site on Google Cloud fetching an RSS feed at its user's request.

Sources

  1. Google: Verify requests from Google crawlers and fetchers · checked 16 Nov 2026
  2. Google: Google's common crawlers · checked 16 Nov 2026
  3. OpenAI: Overview of OpenAI crawlers · checked 16 Nov 2026
  4. Anthropic: Does Anthropic crawl data from the web? · checked 16 Nov 2026
  5. Perplexity: Perplexity crawlers · checked 16 Nov 2026
  6. Bing Webmaster Blog: How to verify that Bingbot is Bingbot · checked 16 Nov 2026
  7. Cloudflare: Web Bot Auth · checked 16 Nov 2026

Written by The Sill Signals Team. Last reviewed 16 Nov 2026.

Share this articleLinkedInXBlueskyEmail

Keep reading

See which AI agents visit your site, and decide what they may do.

Scan your site free