Crawler
A crawler (or spider) is an automated tool that follows links to fetch web content — used by search engines, AI training, price trackers and scrapers.
Control Crawler Access
Allow good bots, block malicious crawlers
What is a Crawler?
A crawler systematically fetches pages and follows the links it finds. Search engines use crawlers to build indexes; archives use them to preserve pages; AI companies use them to collect training data and to retrieve pages when answering a question.
A crawler is a bot, and most crawling is wanted. The distinction worth making is not crawler versus not, but which crawlers you want and on what terms.
Who is crawling now
| Crawler | Purpose | Identifiable by |
|---|---|---|
| Googlebot, Bingbot | Search indexing | Published IP ranges, reverse DNS |
| GPTBot, ClaudeBot, Bytespider | Collecting training data | Published ranges for some vendors |
| OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot | Fetching a page to answer a live question | Published ranges for some vendors |
| Archive and research crawlers | Preservation, academic datasets | Varies; often honest about it |
| Scrapers wearing a crawler's name | Taking your data | Nothing. The name is a disguise |
The split between the second and third rows is the one to get right, because the decision differs. Blocking training crawlers keeps you out of future model weights. Blocking retrieval agents removes you from answers being written today, which is almost certainly not what you want.
Verification, not user-agent strings
A user-agent is a string the client chooses, so it proves nothing. Impersonating a well-known crawler is one of the most common techniques there is — in our own logs, the majority of traffic presenting AI crawler user-agents came from addresses belonging to none of the vendors named, and a substantial share of it was simultaneously probing for /.env and /.ssh/config.
So verify by address. Google, OpenAI, Perplexity and DuckDuckGo publish the ranges their crawlers use; others can be checked by reverse DNS. Anything claiming a crawler identity from outside the published ranges is not that crawler, and should be treated as what it is.
Managing crawler load
- robots.txt is honoured by the crawlers you want and ignored by the ones you do not, which makes it both a useful control and a useful test — a client that reads your robots.txt and then requests the path it disallows has told you something.
- Crawl-delay and sitemap hygiene reduce load from well-behaved crawlers more effectively than blocking.
- Watch the expensive paths. A crawler following every faceted-search permutation can generate substantial load without meaning any harm.
- Do not block the retrieval agents unless you have decided you want to be absent from AI answers. That is a strategy decision, not an infrastructure one.
For bulk extraction dressed as crawling, the controls that work are the ones in web scraping — coherence, behaviour, and making requests cost something.