Web Bot Auth: signed agents solve identity, not intent
Web Bot Auth lets an AI agent cryptographically sign its requests so you can verify who it is. That fixes user-agent spoofing and leaves the harder question untouched: a verified agent can still be doing something you do not want.
For as long as there have been crawlers, the way a well-behaved bot identifies itself has been to write its name in the user-agent string and hope you believe it. Verification, if you bother, means a reverse DNS lookup or checking the request against a published IP range that somebody at the vendor has to keep current.
That was adequate when the well-behaved bots were a handful of search engines. It is not adequate now, and Web Bot Auth is the standards work to replace it.
What it does
Web Bot Auth is an IETF draft that has an automated client sign its HTTP requests using HTTP Message Signatures. The origin verifies the signature against the agent's public key, and gets a cryptographic answer to the question of which agent sent the request.
The immediate benefits:
Spoofing stops working. Anybody can put Googlebot in a user-agent header. Nobody can forge a signature without the key.
IP allow-lists become unnecessary. No more fetching and diffing published CIDR ranges, no more reverse DNS on the hot path, no more stale entries quietly blocking a partner.
Attribution survives infrastructure changes. An agent that moves hosting keeps its identity, because the identity is the key rather than the address.
If your current agent policy is a regular expression over user-agent strings, this is a clear improvement.
What it does not do
A signature tells you who. It does not tell you what for.
Consider what a verified agent can still be doing. It can request your entire product catalogue at a rate nobody agreed to, and be exactly the agent it claims to be while doing it. It can be a general-purpose browsing agent driven by a person whose intent is to scrape you. It can be a legitimate agent whose key has been stolen. It can be an agent that behaved well for six months and then changed what it does, because the company operating it changed its business model.
None of those are identity failures. All of them are trust failures, and a signature has nothing to say about any of them.
The web learned the same lesson about TLS. A padlock means the connection is to the party named in the certificate. It never meant the party was trustworthy, and a great deal of phishing was built on people believing it did.
Identity is an input, not a verdict
Web Bot Auth gives you a better input to a decision you still have to make.
Before: an unverifiable claim of identity, plus behavioural signals.
After: a verifiable identity, plus behavioural signals.
The second is better, because one unreliable part has become a reliable one. There are no fewer moving parts. What you do with a verified identity is still a policy question:
- Is this agent allowed here at all?
- At what rate?
- On which endpoints?
- Does it get the same data a human gets, or a subset?
- What happens when its behaviour changes?
Those are the questions Forrester's Q2 2026 Wave is pointing at when it says the market moves from security first to trust first. Trust is ongoing. Identity is a single fact.
Where this leaves detection
Signed agents will cover the large, reputable, commercially motivated operators who want to be identified, because being blocked costs them customers. That is a useful group, and agent policy for them becomes much simpler.
It will not cover the operators our customers struggle with. A scalper toolkit is not going to sign its requests. A scraper reselling a competitor's free tier is not going to sign its requests. A solver service is not going to sign anything. Those clients will keep doing what they do now: run a real browser build, route through residential proxies, and try to look like a person.
Detecting them still takes the same detailed work: TLS fingerprint mismatches, header combinations real browsers do not emit, and timing signatures that do not match the hardware the client claims. We wrote about why the second half of that is harder to fake than it looks in faking browsers is easy, but not at scale.
So the likely end state is a split policy. Signed agents get an identity-based decision. Everything else gets scored on behaviour. Most sites will need both, and the two answers should end up in the same rules engine rather than in two systems that disagree.
What to do now
If agent traffic is growing on your site, a practical sequence:
Measure it before you police it. Find out what share of your traffic declares itself an agent, what share you believe, and what those clients do. Few organisations have measured it.
Write the policy in terms of endpoints, not clients. Which parts of your site are you happy for an agent to read? Catalogue pages, probably. Checkout, probably not. That question is answerable today and does not depend on any standard shipping.
Do not wait for the standard to start. Web Bot Auth is a draft. Meanwhile, agent traffic is arriving now, and Access Rules on ASN, JA4, headers and path will let you express most of an agent policy today.
When it ships, treat a signature as a strong signal rather than a free pass. A verified agent that starts behaving differently should still trip a detector. If your vendor's answer is that signed agents skip detection entirely, ask what happens when a key leaks.
Related
Working out an agent policy?
If agent traffic is growing on your site and you are trying to decide what to allow, tell us what you are seeing. We will look at the traffic before we reply.
Frequently Asked Questions
What is Web Bot Auth?
Web Bot Auth is an IETF draft specification that lets an automated client cryptographically sign its HTTP requests, using HTTP Message Signatures, so the origin server can verify which agent sent them. It replaces the current situation, where a bot declares itself through a user-agent string that anybody can copy and an IP range that has to be published and maintained out of band.
What problem does Web Bot Auth solve?
Impersonation of well-behaved bots. Today, if you want to allow Googlebot or an AI shopping agent through, you match on a user-agent string and verify it against a published IP range. Both are weak: the string is trivially spoofed and the IP list goes stale. A signature proves the request came from the holder of a specific key, verifiable in the request path without a reverse DNS lookup.
Does Web Bot Auth mean I can stop doing bot detection?
No. A signature answers who, and it does not answer what for. A verified agent can still scrape your catalogue at a rate you did not agree to, or be driven by a user with bad intent, or have its credentials stolen. Identity is an input to a trust decision, not the decision itself. You still need to score behaviour.
What is the difference between Web Bot Auth and signed agents?
Signed agents is the general idea and the name Cloudflare uses for its implementation of it. Web Bot Auth is the IETF specification work underneath. In practice people use both terms for the same thing: an automated client proving its identity with a cryptographic signature rather than a declared string.
How does Web Bot Auth relate to Private Access Tokens?
They point in opposite directions. Private Access Tokens let a client prove a human is present without revealing who that human is. Web Bot Auth lets an automated client prove which bot it is without pretending to be human. One is anonymous proof of humanity, the other is attributed proof of automation. A complete agent policy will eventually use both.