How to block bots on your website: what each method stops, and what gets past it

Blocking bots means refusing requests from software you do not want on your site: scrapers, credential-stuffing tools, scalpers' checkout scripts, spam bots and, if you choose, AI crawlers. It does not mean blocking all automation. Search engines, uptime monitors and some AI assistants are bots you want, and bad bots made up 37% of all internet traffic in 2024 according to Imperva's 2025 Bad Bot Report, so the job is sorting the two rather than shutting the door.

This guide goes through the usual methods in order of effort, says what each one stops, and explains what gets past it.

Decide which bots you want first

Every blocking decision depends on this list, so write it down before changing anything.

How to detect AI agents lists the main AI crawlers and agents and how to verify each.

robots.txt asks crawlers to stay away, but does not block them

A robots.txt file at the root of your site tells crawlers which paths they may fetch. The crawlers that follow it are the ones run by companies with a reputation to keep: search engines and the main AI companies. To ask the main AI training crawlers to stay away from your whole site:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Google-Extended is a token rather than a crawler: Google reads it to decide whether pages Googlebot fetched may be used for Gemini, so disallowing it does not affect search. Our own logs show AI crawlers read robots.txt far more often than any other file meant for them, as our llms.txt study found.

robots.txt does nothing against a bot that chooses to ignore it, and the bots you most want to stop are exactly those. It is the polite first step, not a defence.

Verify any crawler you allow

A user agent is a line of text the client chooses, so a scraper can call itself Googlebot or GPTBot. Before you allow a crawler by name, confirm the request really comes from its operator:

Allow requests whose name and address both match, and block any that use a crawler's name from an address outside its list. Borrowing a well-known crawler's name is common: between 5 and 18 October 2026, 53% of the 17,935 requests to prosopo.io that carried an AI crawler's user agent came from addresses that did not belong to the company named, as how to detect AI agents sets out.

AI agents that browse on a person's behalf, such as ChatGPT Agent, are harder, because many run a real browser from addresses that no published list covers. How they behave on the page is a better guide than where they connect from. An independent study of scroll behaviour, which was discussed on Hacker News, measured how bursty the scrolling of people and AI browsing agents was in the public dataset from the FP-Agent paper. People scrolled in bursts and agents did not, and a model using scroll timing alone told them apart 73.4% of the time, confusing people mainly with ChatGPT Agent. Scrolling on its own is not enough, but it shows why behaviour belongs in the decision.

Server, WAF and CDN rules stop the obvious traffic

Rules on your web server, web application firewall or CDN refuse requests before your application handles them.

These are cheap and worth having, but they work on things an attacker controls, so check what a rule would catch before you enforce it.

A CAPTCHA protects your forms, and a good one spots solving services

A CAPTCHA checks each visitor before a form is accepted, and a modern one lets most real visitors through without showing them a puzzle. It stops most automated form spam on its own. Attackers going after logins, sign-ups and checkouts sometimes pay CAPTCHA solving services to pass the challenge for them using people or software, so a CAPTCHA that only checks whether the challenge was answered can be beaten. A CAPTCHA that also checks who solved it can catch them, because solving services leave traces:

Proof-of-work challenges add a cost to each attempt and work best alongside these checks.

A CAPTCHA protects the forms it is placed on. Scrapers and scripts that call your API directly never load a form, so those routes need checks at the edge as well.

What serious bot detection looks at

Bots that get past the methods above usually run a real browser engine, often a commercial anti-detect browser that has been modified to remove the traces automation normally leaves, and they connect through residential or mobile addresses. Telling them apart from customers takes evidence from several layers at once:

Two of these are worth a closer look. Residential proxies make the address weaker than it looks: in one attack we measured, 131,819 residential IP addresses sent 1.4 million attack requests, but all of them came from just 8 TLS fingerprints and 14 user agents. And hardware is hard to disguise: SIMD instruction timing is one example, because a modified browser can report any number of processor cores but cannot change how long the processor takes to run the same instruction, as we explain in faking browsers is easy, but not at scale. Behaviour can also separate people from AI agents, as the scroll study above found.

No one of these is reliable on its own, and a well-built bot can make any single signal look right. The strength is in checking that they agree. A browser that claims one operating system but connects like another, or claims a phone but times like a server, is hard to fake consistently, because the network stack and the hardware are not something a modified browser can change. Checks that read only what JavaScript reports lose to browsers patched to report whatever the attacker likes.

Where to run the checks

Run checks at your CDN edge if you can, in front of every page and API endpoint, so a request is refused before it reaches your servers and costs you compute. Checks inside a form widget only see the visitors who reach that form.

Start in a preview mode that records what would be blocked without blocking it. A week or two of real traffic shows which rules catch customers, and it is much easier to fix those before they cost you a sale. Challenge visitors you are unsure about rather than blocking them, and keep a readable reason for every block, because the first time a real customer is refused, someone will need to explain why.

Bot blocking checklist

1. List the bots you want, and allow them by verified identity, not by name.
   • Include search engines, the AI crawlers and assistants you want citing you, uptime monitors, payment callbacks and partner integrations.
   • Record how each one is verified: reverse DNS for Googlebot and Bingbot, published IP ranges for the AI crawlers.
2. Use robots.txt to set your policy for search and AI crawlers.
   • Add a User-agent and Disallow block for each AI crawler you do not want, such as GPTBot, ClaudeBot and CCBot, and the Google-Extended token for Gemini training.
   • Leave search crawlers alone unless you mean to drop out of search.
3. Block requests that use a known crawler's name from outside its published addresses.
   • Refresh the vendors' IP range files regularly, because they change.
   • Log what this rule blocks for a week before enforcing it, to catch any proxy or CDN in front of the real crawler.
4. Rate-limit per endpoint, with tighter limits on login, sign-up, password reset and checkout.
   • Count per account, session and device as well as per IP address, so a bot rotating proxy addresses still hits a limit.
   • Rate-limit failed logins across the whole endpoint, not only per account.
5. Put a CAPTCHA on the forms that matter, choosing one that checks who solved the challenge as well as the answer.
   • Verify every token on your server, and make sure each one can be used once and expires quickly.
   • Pass the visitor's IP address with the token when you verify it, so a token solved somewhere else can be refused.
6. For pages and APIs bots target directly, add detection at the edge that compares network, device and behaviour evidence.
   • Cover the API routes your apps call as well as your web pages.
   • Decide per route whether a doubtful request is challenged or blocked.
7. Run new rules in preview first, and record a reason for every block.
   • Review what a rule would have caught before it blocks anything, and look for real customers among it.
   • Keep the reason with each blocked request, so support can answer a customer who was refused.

How Prosopo blocks bots

Prosopo Protect runs at your edge on Cloudflare Workers or AWS Lambda@Edge, in front of every page and API you choose. It checks the network, device and behaviour evidence described above on each request, allows verified good bots, and lets you set rules for AI agents. It starts with 14 days in preview mode, so you see what it would block on your own traffic before it blocks anything, and every block carries a reason your team can read. Pricing is published on the pricing page, with traffic not metered.

For forms, Procaptcha checks the same network, device and behaviour evidence as well as the answer, and its IP validation rules can reject a token used from a different country, city or network from the one that solved it. It is free for 10,000 requests a month. For particular attacks, see scraping, credential stuffing and spam bot protection.