Gray bots: how to decide which AI crawlers and agents to allow

Gray bots are automated visitors that are neither clearly good nor clearly bad: AI crawlers, assistants fetching pages and personal agents. What Gartner means by the term, and a policy for each part of your site.

Chris Taylor
Chris Taylor

8 min read

Gray bots are automated visitors that are neither clearly good nor clearly bad. An AI crawler collecting your pages, an assistant fetching one to answer a question, and a personal agent buying something for a customer are all automated, and each can be wanted on one part of your site and unwanted on another. Gartner put the term at the centre of its first Market Overview for Bot Protection and Management Solutions in November 2026. This page explains what counts as a gray bot, how to check one is genuine, and how to write a policy for each.

Where the term comes from

Barracuda used "gray bots" in April 2025 for generative-AI scrapers that are not openly malicious but collect content in bulk. One web application it tracked received 9.7 million requests from these bots in 30 days, and another received more than half a million in a single day. ClaudeBot was the most active, followed by Bytespider, PerplexityBot and DeepSeekBot.

Gartner widened the meaning. Its Market Overview describes gray bots as bots that "cannot be clearly classified as good or bad without evaluating their intention and business relevance". That covers AI assistants and agents visiting on behalf of a person, as well as scrapers.

Akif Khan, the report's co-author, listed the traffic types a business now has to tell apart: humans, search engine indexing bots, AI platform crawlers collecting content, AI platforms fetching a page to answer a user's question, and personal agents sent to carry out an action. The last three are gray bots. Khan says most businesses have not yet decided on a policy for the last one.

Five kinds of gray bot

KindExamplesSays who it is?How to verify itWhat it does for you
AI training crawlerGPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagentYes, in the user-agentVendor's published IP ranges; Meta by its network, AS32934Nothing directly. Your content goes into a model
AI search crawlerOAI-SearchBot, Claude-SearchBot, PerplexityBotYesVendor's published IP rangesPuts you in the assistant's search index, so it can cite you
Answer-time fetcherChatGPT-User, Claude-User, Perplexity-UserYesVendor's published IP rangesA person asked about you, and the assistant is reading your page to answer
Personal agentMeta's Muse, OpenAI's Dots, ChatGPT agent, agentic browsers such as Perplexity CometNot in the user-agent. Most look like Chrome, though some sign their requestsRequest signatures where sent (Web Bot Auth); otherwise behaviour and device signalsOften a real customer getting something done
Commercial scraper or aggregatorPrice, review and travel aggregatorsSometimesPublished ranges if any, or behaviourA listing elsewhere, or a competitor with your prices

The first three declare themselves and can be checked against lists the vendors publish. The fourth names itself only through a signature, when it sends one. The ChatGPT agent and Meta's Muse both browse as an ordinary desktop Chrome on Linux, and the signature is the only thing that says who they are. We go through how to identify each one in how to detect AI agents on your website.

Impostors are not gray bots

A request that names itself GPTBot or Claude-User but comes from an address the vendor does not use is not a gray bot. It is a bad bot using a gray bot's name, usually because it expects to be allowed.

This is common. We count it on our own site, and in 45 days of logs to 18 November 2026, 48% of 84,321 requests carrying an AI user-agent came from addresses that did not belong to the vendor named. For OpenAI's names it was worse: only 20% of the 12,807 requests calling themselves GPTBot, OAI-SearchBot, ChatGPT-User or OAI-AdsBot came from OpenAI.

The genuine ChatGPT agent never uses those names. Across 25.2 million captcha sessions on our customers' sites over 30 days, it appeared 1,599 times, always as an ordinary Chrome on Linux and identified only by a valid chatgpt.com signature. Only 142 unsigned sessions carried an OpenAI name at all. The full breakdown, including Muse, Google-Agent and DuckDuckGo, is in signed agents in 25 million captcha sessions.

So the first rule of any gray bot policy is to verify before you allow. An allow rule that matches on the user-agent alone is an allow rule for every scraper that copies it, and it still misses the signed agents, which do not put their name in the user-agent.

A gray bot policy for each part of the site

The useful question is not "is this a bot?" but "what is this visitor doing, and on which page?". The same assistant can be welcome on a product page and unwelcome at checkout. A policy works best written as a grid:

Part of the siteTraining crawlerAI search crawler and answer-time fetcherPersonal agentUnverified bot claiming an AI name
Public articles, docs, help pagesYour choice: allow, or refuse in robots.txt and enforce by IPAllowAllowBlock
Product, listing and search pagesYour choice; rate-limit if it is heavyAllowAllowBlock
Prices, stock levels and bulk dataBlock or rate-limitAllow single pages; rate-limit bulk readsAllow at human paceBlock
Signup and loginBlockBlockRequire evidence of a personBlock
Checkout on limited stock, ticket purchaseBlockBlockRequire evidence of a person, with a limit per accountBlock
Password changes, payouts, bulk exportBlockBlockRequire evidence of a personBlock
Public APIPer your API termsPer your API termsAuthenticated keys onlyBlock

Three things decide most cells.

What the action costs if it happens ten thousand times. Reading a page costs you almost nothing. Creating an account, holding a ticket or exporting a customer list costs a lot. Put the human check where the cost is.

Whether the visitor brings anything back. Answer-time fetchers are how assistants cite you, and blocking them takes you out of the answers. On our own site, verified AI fetches outnumber click-throughs from assistants by about 120 to one, so most of the value is in being read, not in the visit. Training crawlers bring nothing back directly, so whether to allow them is a decision about your content, not about security.

Whether a person stands behind it. A personal agent finishing a task for a customer is a customer. You do not need to block it, but you do need the customer to complete the step that matters themselves. That limits a scalper's tool in the same place, whatever software it runs on.

Personal agents: the row most sites have not decided

Personal agents are the hardest row because they do two things at once. They carry out a real customer's request, and they carry it out with the same automation a scalper or a fake-account farm uses.

Signing helps with identity but not with intent. Web Bot Auth lets an agent sign its requests so you can prove which agent sent them. A signed agent can still buy all your stock or sign up a thousand accounts if the person driving it wants to. And the agents that do not sign, which still includes some of Muse's traffic, have to be judged on behaviour, device and network signals anyway.

Most sites have not decided yet. In our captcha data, only 8% of signed ChatGPT agent sessions skipped the captcha, on sites that had chosen to let verified agents through, and no Muse session did. Everywhere else a signed agent was scored like any other visitor. That is the gap Gartner's Akif Khan describes: the signature is there, and the policy for what it should change is not.

So treat a personal agent the way you would treat a customer using a password manager or a browser extension: let it fill in the form, and make sure a person is present for the step that costs money. We set this out in more detail in AI agent identity.

What to ask a bot management vendor about gray bots

Gartner predicts that by 2028, 80% of enterprises with critical public-facing web applications will adopt a bot protection and management solution. If gray bots are part of why you are looking, ask each vendor:

  1. How do you verify an AI crawler or fetcher? By published IP ranges and signatures, or by user-agent only?
  2. Can I set a different policy for each part of the site? Allow agents on product pages and require a person at checkout, without writing code.
  3. What happens to a personal agent you cannot identify? Is it blocked, challenged or allowed, and can I change that?
  4. What do you tell me when you block one? A reason I can read and pass to a customer, or a score?
  5. Can I run it without blocking first? So I can see which gray bots reach which pages before I enforce anything.

How Prosopo Protect handles gray bots

Prosopo Protect is a script tag on your pages, with edge protection on Cloudflare Workers or AWS Lambda@Edge, and gives every request an allow, block or challenge verdict.

  • Impostors are blocked. Prosopo manages the detection rules, including checks that compare the TCP and TLS connection with the device the browser claims to be. You can add your own Access Rules on IP range, ASN, user-agent, JA4 fingerprint and IP category, so an AI user-agent from a data-centre address that is not the vendor's can be refused while the genuine fetcher is let through.
  • Policy per path. Browsing stays open, while checkout, signup and account changes require evidence of a human.
  • A reason on every block. Each block names the detector that fired, so when a customer's agent is stopped you can tell them why.
  • Preview mode first. Every deployment starts with 14 days where everything is checked and nothing is blocked, so you see which gray bots reach which pages before you enforce.

Protect starts from $1,990 a month, priced by your number of sites, SLA and data requirements rather than per request, within fair use.

Tagged gray-bots ai-agents ai-crawlers agentic-browsers bot-protection-and-management bot-and-agent-trust bot-management gartner prosopo

Chris Taylor

Chris Taylor

Building privacy-first bot protection at Prosopo.

More articles by Chris Taylor →

Writing a policy for AI crawlers and agents?

Leave your work email and an engineer will send you the steps, effort and cost of moving to Prosopo for your stack, usually within one working day.

Work email only. We use it to answer this request. See our privacy policy.

Prefer a call? Book an assessment

Questions

Something else? Ask an engineer.

What are gray bots?

Gray bots are automated visitors that cannot be called good or bad until you know what they want and what they are worth to you. They include AI training crawlers such as GPTBot and ClaudeBot, AI assistants fetching a page to answer a user's question, personal agents such as Meta's Muse and OpenAI's Dots acting for a person, and commercial scrapers and aggregators. Gartner's 2026 Market Overview for Bot Protection and Management Solutions describes them as bots that "cannot be clearly classified as good or bad without evaluating their intention and business relevance".

Who came up with the term gray bots?

Barracuda used it in April 2025 for generative-AI scraper bots that are not openly malicious but collect large volumes of content. In one application Barracuda tracked, these bots made 9.7 million requests in 30 days. Gartner adopted the term in its first Market Overview for Bot Protection and Management Solutions in November 2026, and applied it more widely to AI agents as well as scrapers.

Is it gray bots or grey bots?

Both spellings mean the same thing. Gartner and Barracuda use gray; British and Australian coverage often writes grey.

What is the difference between good bots, bad bots and gray bots?

A good bot is one almost every site wants, such as a verified Googlebot. A bad bot is one almost no site wants, such as a credential-stuffing tool or a scalper. A gray bot is one where the answer depends on the site and the page: an AI assistant fetching your product page to cite you is useful, while the same assistant placing a hundred orders on limited stock is not.

Should I block gray bots?

Not as a group. Block anything that claims to be a known AI bot but comes from addresses the vendor does not use, because it is lying about who it is. For the rest, decide per type and per part of the site: let assistants read public pages so they can cite you, choose whether training crawlers may collect your content, and require evidence of a human on steps that cost money, such as checkout on limited stock, signup and bulk export.

How do I handle personal agents like Meta's Muse or OpenAI's Dots?

Let them browse and compare, and ask for the person on the steps where a mistake or abuse costs you money. The ChatGPT agent and, since November 2026, some of Muse's traffic sign their requests with Web Bot Auth, which proves who sent them but not why. Neither says who it is in the user-agent: both look like an ordinary Chrome on Linux. Unsigned personal agents can only be identified by behaviour and device signals.

How do I verify a gray bot is who it says it is?

Check the source address against the IP ranges the vendor publishes, or check the request signature where the agent sends one. OpenAI, Anthropic, Perplexity and DuckDuckGo publish range files for their crawlers and fetchers. Meta publishes no list, so check that the address belongs to Meta's own network, AS32934. In 45 days of our own logs, 48% of requests using an AI bot's name came from addresses that did not belong to that vendor, and only 20% of requests using an OpenAI name came from OpenAI.

Gartner Market Overview for Bot Protection and Management Solutions 2026: what it says Gartner's first Market Overview for Bot Protection and Management Solutions, November 2026: gray bots, the 80% by 2028 prediction, the vendors named, and how it compares with the Forrester Wave and Cloud WAAP research. Web Bot Auth: signed agents solve identity, not intent Web Bot Auth lets an AI agent cryptographically sign its requests so you can verify who it is. That fixes user-agent spoofing and leaves the harder question untouched: a verified agent can still be doing something you do not want. Agentic browsers: your customer is now a robot, and that is fine An agentic browser is automation operated by a real customer. It fails every classic bot test and still deserves to reach your checkout. The answer is to require evidence of a human on particular steps, rather than to judge the client. llms.txt: AI crawlers read robots.txt 170 times as often We have published an llms.txt since May 2026. Over twelve days of IP-verified logs, AI crawlers fetched our robots.txt 511 times and llms.txt three times. x402 and agent payments: what a machine-payable endpoint means for your site x402 revives HTTP 402 so an agent can pay for a request in stablecoins and retry it. That turns abuse into a priced transaction, and it does not decide who is allowed to buy. MCP security: the five ways agent tool access goes wrong MCP turns tool descriptions into model context, which makes the integration layer an attack surface. Tool poisoning, confused deputy, token passthrough, SSRF and rogue servers are the five that matter.