How to detect AI agents on your website

"AI agent" covers four kinds of traffic that behave differently and need different checks. Three of them declare themselves and can be verified against lists the vendors publish. The fourth, the agentic browser, runs a real Chrome on a real customer's machine and usually declares nothing at all.

The other thing to know before you start is that the name on the request is often a lie. In our own logs, more than half of the requests calling themselves AI agents were not.

Four kinds of AI agent traffic

Training crawlers collect pages that may be used to train a model. GPTBot (OpenAI) and ClaudeBot (Anthropic) are the best known. They crawl on their own schedule and respect robots.txt.

Search crawlers build the index an assistant searches when it answers a question. OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot are examples. Blocking them keeps you out of that assistant's search results.

User-triggered fetchers visit a page because a person has just asked an assistant something that needs it. ChatGPT-User, Claude-User and Perplexity-User work this way, and so does DuckAssistBot. OpenAI and Perplexity both say that because a person started the fetch, robots.txt rules may not apply.

Agentic browsers carry out a task for a person — finding a product, filling in a form, booking something — by driving a browser. Some run in the vendor's cloud, such as ChatGPT's agent. Others run on the user's own computer, such as Perplexity Comet. We cover why they fail every classic bot test in agentic browsers.

Known AI agents and how to verify each

These are taken from each vendor's own documentation, checked on 2 November 2026.

Sources: OpenAI's crawler documentation, OpenAI's agent allowlisting guide, Anthropic's crawler documentation, Perplexity's crawler documentation, DuckDuckGo's DuckAssistBot page and Google's crawler list. On Comet, both 4sysops and the MIT AI Agent Index report that it uses the same user-agent as Chrome, and Perplexity does not document one.

More than half the AI user-agents in our logs were spoofed

Between 5 and 18 October 2026, prosopo.io received 17,935 requests carrying an AI user-agent. 53% of them came from IP addresses that did not belong to the vendor named.

Most of the impostors were on Google Cloud. Single addresses presented 11 to 13 different AI user-agents each and used them to probe paths such as /.env and /actuator, which is somebody looking for leaked credentials and exposed admin endpoints, not an assistant reading a blog post. An AI name is a cheap disguise on any site that lets AI agents through on the user-agent alone.

For some vendors there was no genuine traffic at all. Every request presenting an xAI, Cohere, Mistral or Google-Extended user-agent came from a spoofer. Google-Extended is the clearest case, because Google never sends it as a user-agent.

The genuine traffic is real and worth having. Between 19 October and 1 November 2026, verified answer-time fetches by AI assistants averaged 248 a day. Those are the requests that put your pages into an assistant's answer, which is why blocking on the user-agent alone costs you something whichever way you get it wrong.

Four ways to identify an AI agent

User-agent string. This is fine for sorting traffic you have already verified, and for writing robots.txt rules, but never treat it as proof, and never block on its absence, because anyone who does not want to be noticed stops sending it.

Published IP ranges. This is the most useful check available today. Match the source address against the vendor's JSON file from the table above, and refresh the files regularly because they change: OpenAI's ChatGPT-User list was regenerated on 27 October 2026 and Perplexity's on 28 October. A request claiming to be GPTBot from an address outside gptbot.json is not GPTBot.

Reverse DNS. Look up the hostname for the source IP, check it belongs to the vendor's domain, then look that hostname up again and check it returns the same IP. Google documents this for its crawlers, whose hostnames end in googlebot.com, in its guide to verifying Google's crawlers. Not every AI vendor supports it, so use the IP files where they exist.

Request signatures. With Web Bot Auth, an agent signs each request using HTTP Message Signatures (RFC 9421), and you check the signature against a key directory the operator publishes. OpenAI's agent already does this. A signature travels with the request wherever it comes from, so it keeps working when an IP list goes stale. It tells you which agent sent the request but not whether you want it to do what it is doing, which we cover in AI agent identity.

Detecting agentic browsers with behaviour and network signals

An agentic browser running on the user's own computer has no IP range to check and often no signature. It is real Chromium on real hardware, on a home connection, carrying the user's cookies.

Checks for automation tools inside the browser are not a durable answer. Commercial anti-detect browsers patch Chromium at the C++ level, so anything visible to JavaScript can be removed or faked. Signals rooted in hardware and the network are harder to change:

• The connection. The TLS handshake (summarised as a JA4 fingerprint) and the TCP behaviour of the connection show what really opened it. If the browser says Windows and the TCP stack is Linux, or the connection arrives through a proxy tunnel, the claim does not hold up.
• The address. IP reputation, the network the address belongs to, and whether it is a known residential proxy exit. A cloud-hosted agent arrives from a data centre, and an impostor rotating through a proxy pool shows up as many addresses sharing one pattern.
• Timing. A person pauses, re-reads and gets distracted, but an agent acts at an even rhythm, and its input lacks the small irregularities of a hand on a mouse or a finger on glass.
• Volume and spread. One customer's agent is one session doing one task. Hundreds of sessions with the same pattern across rotating addresses is an operator using agent tooling at scale.

Blocking, allowing, or requiring a human on specific steps

Once you can tell the kinds apart, the policy is a business decision rather than a technical one.

Block impostors outright. A request with an AI user-agent from an address the vendor does not publish is lying about who it is, and nothing is lost by refusing it.

Decide training crawlers on their own terms. If you do not want your content used for training, say so in robots.txt and enforce it by IP for the crawlers that publish ranges.

Let search and answer-time fetchers through. They are how assistants find and cite you. In our llms.txt experiment, IP-verified AI crawlers fetched robots.txt about 170 times as often as llms.txt, so the effort belongs in the pages themselves.

Require evidence of a human on the steps that cost you money. Let any agent browse, search and compare, and ask for a person at checkout on limited stock, account signup, password changes and bulk export. The customer who delegated the task finishes that step themselves, and a scalper or scraper is restricted in the same place whatever tool they are using. If you expose tools to agents directly, the same thinking applies to MCP servers.

Prosopo Protect runs at your edge on Cloudflare Workers or AWS Lambda@Edge and checks every request. Access Rules match on IP range, ASN, user-agent, JA4 fingerprint and IP category, so a rule can refuse AI user-agents arriving from data-centre addresses that are not the vendor's. Network checks compare the TCP connection with the device the browser claims to be, and every block names the detector that fired, so when a real customer's agent is stopped you can tell them why.