Definition
A user agent is the software making a web request, and the User-Agent is the HTTP header where it names itself. The string can include the app, its version, the operating system and the vendor. Any client can send any value, so it is a claim, not proof.
Why it matters
The User-Agent is the first thing most site owners look at when they ask "who is visiting?" Your analytics, server logs and robots.txt rules all use it. A rule like "allow GPTBot" matches on this string.
But the header is set by the visitor. MDN notes that any client can send any value, and tools such as curl send whatever you tell them. So a request that says it is a well-known bot may not be. Treat it as a hint.
How it works
A browser sends a long string that begins with a historical Mozilla/5.0 token, followed by details about the platform and engine. Browsers now shorten some of these details for privacy. Crawlers usually send something simpler, often with a link to a page about the bot.
To confirm a claim, check something the visitor can't easily fake. Google, for example, tells site owners to run a reverse DNS lookup on the IP address or match it against its published IP ranges. See verified bots.
Example
Your logs show this:
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
That request says it is Googlebot. Whether it is depends on the IP address behind it. If the IP fails Google's check, call it a request that claims to be Googlebot but fails verification. It is an unconfirmed claim, so don't give it special access. The rise of agentic commerce makes this check more important.
Sources
- MDN: User-Agent header · checked 15 Nov 2026
- Google Search Central: Verifying Googlebot and other Google crawlers · checked 15 Nov 2026