Scraping Protection: Stop Web Scrapers and Spoofed AI Crawlers

Who scrapes websites now

Scraping used to mean a competitor copying your prices or a script lifting your listings. Those still happen, but four kinds of scraper now arrive, and they need different answers.

AI training crawlers. Bots such as OpenAI's GPTBot collect pages that may be used to train AI models. The large operators name their crawlers, publish the IP addresses they use, and say they follow robots.txt.

AI answer engines fetching live. When someone asks ChatGPT, Claude or Perplexity a question, the assistant may fetch your page there and then to answer it. OpenAI's bot documentation says that because a person starts these requests, "robots.txt rules may not apply" to ChatGPT-User. This traffic is often traffic you want, because it is how your pages end up quoted in an answer. Between 19 October and 1 November 2026, verified answer-time fetches by AI assistants to prosopo.io averaged 248 a day.

Price, content and data scrapers. Competitors, aggregators and resellers who want your catalogue, your prices, your listings or the results of your search. They do not announce themselves.

Scrapers using residential proxies and anti-detect browsers. The serious operators send each request through a different home broadband connection and run a real browser that has been altered to report a believable device. Each request looks like a separate ordinary visitor, and nothing about the rate stands out. AI agents that drive a real browser, which we cover in agentic browsers, look much the same from the server.

robots.txt is a request, not a block

robots.txt tells crawlers which parts of a site you would like them to leave alone. The standard, RFC 9309, says plainly that "these rules are not a form of access authorization". A crawler that obeys it does so voluntarily.

It is still worth maintaining, because the crawlers that behave really do read it. Between 6 and 17 October 2026, verified AI crawlers fetched our robots.txt 511 times and our llms.txt three times. But a scraper that wants your content ignores the file, and an assistant fetching a page for a person may not consult it either.

More than half the AI user agents in our logs were spoofed

Blocking or allowing by user agent assumes the client is telling the truth. Between 5 and 18 October 2026, prosopo.io received 17,935 requests carrying an AI user agent, and 53% of them came from IP addresses that did not belong to the vendor named. Most of the impostors were on Google Cloud, and single addresses presented 11 to 13 different AI user agents each.

So the user agent cannot be trusted whether you allow or block by it. If you let anything calling itself GPTBot or ClaudeBot through, scrapers will use those names. If you block by AI user agent, you stop only the crawlers that were honest about who they are, while scrapers carry on with an ordinary Chrome user agent. We go through the checks for each AI agent in how to detect AI agents.

Which crawlers to allow and which to stop

Start from what you want, then verify it by something a scraper cannot copy.

Allow verified search engines and AI assistants. Google, OpenAI, Anthropic, Perplexity and others publish the IP ranges their crawlers use. A request that claims to be one of their crawlers and comes from inside the published range is genuine, and one that comes from anywhere else is an impostor. Some agents can also sign each request using Web Bot Auth, which proves who sent it without relying on an IP list, but a signature tells you who the agent is and not what it is doing, so a verified agent can still take more than you intended.

Decide on AI training crawlers as a business question. Whether you want your content used to train models is up to you. The named training crawlers can be refused in robots.txt and, if you want certainty, with a block rule that matches their user agent and published IP range together.

Stop everything else that behaves like a scraper. That means automation with no verified identity that is reading pages, prices or API responses at scale, whatever user agent it sends and whatever network it comes from.

How Prosopo Protect stops scrapers

Prosopo distinguishes legitimate visitors, search crawlers and authorised AI agents from scraping infrastructure on every request, and you set the policy rather than choosing between allowing all automation and blocking all of it.

• How the visitor behaves. Mouse movement, scroll rhythm and typing cadence. Automation repeats itself in ways a person does not, and that holds even when every other detail about the session looks ordinary.
• Where the traffic really comes from. Attackers route through home broadband connections and racks of real phones so each request looks like a separate ordinary visitor. Residential proxy detection checks the TCP connection for signs of a relay, scores every request against IP intelligence, and matches exit addresses against the proxy networks we rent and probe ourselves.
• Browsers that contradict themselves. An altered browser reports one device and behaves like another. Named detectors such as Inconsistent hardware readings for device and HTTP header fingerprint mismatches catch the contradiction.
• Sudden surges from the wrong places. A normal day sees steady traffic from your usual markets. A coordinated burst from hosting networks or unfamiliar regions gets extra checks, without slowing anyone else down.
• Your own allow and block rules. Access Rules match on IP range, ASN, country, user agent, JA4 TLS fingerprint and IP category (VPN, proxy, datacentre), alone or combined. Allow a crawler by its user agent and published IP range together, block hosting networks a scraper uses, or send suspicious traffic to an image, puzzle or proof-of-work challenge. Rules take effect the moment they are saved.
• Pages and APIs handled differently. Blocked HTML pages get a branded interstitial, and JSON APIs get the right HTTP status and an X-Prosopo-Decision header, so scraping through your API is stopped as well as scraping of your pages.

Every block names the reason it was made, and Protect's edge protection runs on Cloudflare Workers or AWS Lambda@Edge, so scraper traffic is stopped before it consumes capacity on your servers.

Choosing scraping protection

Check these before you compare prices:

• APIs as well as pages. Scrapers move to your JSON API as soon as the pages are protected. Protection that only covers HTML pages leaves the API unprotected.
• Verified crawlers allowed by address, not by name. Search engines and AI assistants are how people find you. Allow them by their published IP ranges, because their user agents are the first thing a scraper copies.
• Residential proxies caught on the connection. IP block-lists do not catch a scraper that rents home broadband addresses. Look for checks on the connection itself.
• A reason on every block. When a real customer is stopped, you need to be able to see why.
• A price that does not rise with the attack. If you pay per request, a scraping surge raises your bill. Prosopo Protect is priced by number of sites rather than per request, within fair use, from $1,990 a month, and starts with 14 days in preview mode so you see what it would block first.

A competitor scraping a search product

A subscription search product had its free tier consumed at scale by a competitor reselling the results. Over thirty days: 7,824,429 sessions scored, 315,766 stopped as automated, and 79% of real visitors cleared without seeing a puzzle. The scraper was running slowly from home broadband addresses, so nothing about its rate stood out. What caught it was the client contradicting itself. The full figures are in the scraped SaaS case study.

Prosopo products for stopping scrapers

• Enterprise (Prosopo Protect) — put bot-aware verification in front of catalogue, search and content APIs.
• Residential Proxy Detection — catch scrapers renting home broadband addresses.
• Access Control — block hosting ASNs, TLS fingerprints and user agents associated with scraping toolchains.
• Bot Protection — surface a 0–1 score per request so your backend can rate-limit or throttle suspicious patterns.