Web Bot Auth: how signed requests verify AI crawlers
Web Bot Auth lets a bot sign its requests so you can verify who sent them without trusting user agents or IP lists. See the headers, key lookup and who signs.
By Crawlito · Updated 9 min read
In short
- Web Bot Auth has a bot sign each request with a private key using RFC 9421 HTTP Message Signatures, and sends Signature, Signature-Input and Signature-Agent headers so a verifier can check it against a published public key.
- A valid signature proves the request came from whoever holds the key, which a spoofable user agent string or a shared IP address cannot.
- Cloudflare, Akamai and AWS WAF Bot Control each document verifying these signatures, and operators such as OpenAI (ChatGPT agent) and Google (some Google-Agent requests) document sending them.
- Most AI crawlers do not document signing at the time of writing: OpenAI's crawler page lists IP ranges for OAI-SearchBot, GPTBot and ChatGPT-User, not signatures.
- The protocol is still an IETF Internet-Draft, so keep IP and reverse DNS checks as a fallback and do not block every unsigned request.
On this page
- Why user agents and IP lists are not enough
- What a signed request looks like
- How the key directory and public key lookup work
- Which platforms verify signed bots, and what they do with the result
- Which crawlers and agents sign requests today
- How to check whether a request is signed
- What to do when a request is not signed
- Status of the standard and limits of relying on it
- Frequently asked questions
- Check what a crawler sees on your page
Web Bot Auth is a way for a bot to prove who it is by signing its HTTP requests with a private key. The site, or the CDN in front of it, fetches the matching public key from a URL the bot publishes and checks the signature. If it verifies, the request came from whoever controls that key, which a user agent string or a shared IP address cannot show. It works today for some agents and CDNs, but it is still an IETF draft and most AI crawlers do not document signing yet.
This guide covers what the headers mean, how the key lookup works, which platforms verify signatures, who signs, and how to see whether any signed request is reaching you. Verifying a bot only tells you who it is. Whether an assistant then cites your pages depends on many things outside this mechanism.
Why user agents and IP lists are not enough
The two traditional checks both have gaps. A user agent is text the client chooses, so anyone can send GPTBot or Googlebot. IP lists are stronger, and OpenAI publishes ranges for OAI-SearchBot, GPTBot and ChatGPT-User, but Cloudflare's original proposal points out that addresses can be shared by many users or services, such as VPNs and shared hosting, and that ranges change as infrastructure moves. Our guide to verifying real AI crawlers covers both methods in detail.
Agents add a new wrinkle. A browsing agent that acts for a person may run from a cloud browser with no fixed user agent, so there is nothing stable to match. Web Bot Auth moves the proof from "what the request says" to "what the sender can sign".
What a signed request looks like
Web Bot Auth builds on RFC 9421, the HTTP Message Signatures standard (a Proposed Standard published in 2024). In RFC 9421, Signature-Input lists which parts of the message are covered and the signature's parameters, and Signature carries the signature value. Web Bot Auth adds a third header, Signature-Agent, and fixes some of the parameters.
Here is an illustrative request. The values are placeholders, not a real key or signature.
GET /pricing HTTP/1.1
Host: example.com
Signature-Agent: "https://signer.example"
Signature-Input: sig1=("@authority" "signature-agent");created=1700000000;expires=1700000060;keyid="JWK-THUMBPRINT";alg="ed25519";tag="web-bot-auth"
Signature: sig1=:BASE64-SIGNATURE:What each part does, according to Cloudflare's Web Bot Auth documentation:
| Part | Meaning |
|---|---|
Signature-Agent | Identifies the signer. Cloudflare requires a double-quoted https:// URL, and requires it to be listed in the Signature-Input components. |
@authority | The host the request targets. Cloudflare says to sign at least this component, which ties the signature to the host it was sent to. |
created, expires | A validity window. Cloudflare suggests an expiry of about a minute. |
keyid | The JWK thumbprint of the signing key. |
tag="web-bot-auth" | Marks the signature as a Web Bot Auth one. |
Signature | The signature itself. |
The header format is still moving. The IETF draft describes Signature-Agent as a dictionary with a label, for example sig="https://signer.example.com", and calls the bare quoted string a legacy form that verifiers may accept. Google's documentation shows the labeled form: g="https://agent.bot.goog". Cloudflare's page, at the time of writing, describes the quoted string. If you write your own verifier, read the current draft rather than copying one example.
How the key directory and public key lookup work
The signer publishes its public keys at a URL a verifier can fetch. The draft puts this at /.well-known/http-message-signatures-directory, served over HTTPS as a JSON Web Key Set with the content type application/http-message-signatures-directory+json. The Signature-Agent value tells the verifier which host to ask.
A verifier then does roughly this:
- Read
Signature-Agentand check it names an operator you recognise, overhttps://. - Fetch that operator's key directory, or use a cached copy. Google says to cache according to the
Cache-Controlheader. - Find the key whose thumbprint matches
keyid. - Rebuild the signed text from the components listed in
Signature-Input, using the request it actually received. - Check the signature with the public key, then check that
createdandexpiresmake the signature currently valid and that@authorityis your own host.
The draft says a directory response only counts on a 200 status and that verifiers must not follow redirects automatically. Cloudflare adds that it does not validate nonces or check for replays, so a captured signature can be reused until it expires. That is why short expiry times matter.
Operators also sign their key directory. Cloudflare's setup steps require the directory to carry Signature and Signature-Input headers, so a verifier can tell the key list is genuine. For ChatGPT's cloud browser, OpenAI's agent access page publishes the directory at https://chatgpt.com/.well-known/http-message-signatures-directory.
Which platforms verify signed bots, and what they do with the result
Three platforms document this verification at the time of writing.
Cloudflare. Web Bot Auth is one of the two validation methods for its verified bots program, the other being IP validation, which needs a published IP list or reverse DNS plus a user agent match pattern. In August 2025 Cloudflare also introduced signed agents, a category for agents that a user directs and that sign with Web Bot Auth. Cloudflare's Web Bot Auth page refers to the cf.bot_management.verified_bot field as one its verification populates, which you can use in rules. Our Cloudflare settings guide shows where those policies live.
Akamai. Akamai's November 2025 post says its edge fetches the key directory and checks the signature. A valid signature classifies the request as validated-bot traffic, and you can then allow it or apply a challenge, delay or block.
AWS WAF Bot Control. AWS says in a July 2026 security blog post that Web Bot Auth support needs rule group version 4.0 or later and that Bot Control allows verified agent traffic by default. It documents labels such as awswaf:managed:aws:bot-control:bot:web_bot_auth:verified, plus invalid, expired and unknown_bot. The older verified bots example covers the general bot:verified label and does not mention Web Bot Auth, so use the blog post for the signature labels. More on Bot Control in AWS WAF Bot Control and AI crawlers.
In all three, the signature feeds a decision you control. AWS describes using the web_bot_auth labels in your own custom rules, and Akamai lets you choose the action for validated-bot traffic.
Which crawlers and agents sign requests today
Operators describe signing for a small set of products, mostly agents rather than classic crawlers. Exa's search crawler is one exception.
- ExaSearchBot. Exa says every ExaSearchBot request is signed, with Ed25519 keys published at
https://crawler.exa.ai/.well-known/http-message-signatures-directory, and that Cloudflare and Akamai validate it automatically. See ExaSearchBot. - ChatGPT agent (cloud browser). OpenAI's agent access page, which now calls it the ChatGPT cloud browser, says its requests carry
Signature,Signature-Inputand aSignature-Agentof"https://chatgpt.com". See ChatGPT agent in server logs. - Google-Agent. Google calls its Web Bot Auth use experimental. Only a subset of Google-Agent requests are signed, under the identity
https://agent.bot.goog, and Google recommends keeping IP, reverse DNS and user agent checks alongside it. Details are in Google-Agent and robots.txt. - Other signed agents. Cloudflare's first signed agent cohort in 2025 also named Goose from Block, Browserbase and Anchor Browser. AWS documents that Amazon Bedrock AgentCore Browser can sign requests, in a feature it labels preview.
- Your own tools. Shopify offers signatures for your own crawlers, scripts or tools that access your public store, with a
Signature-Agentof"https://shopify.com".
What is not documented matters just as much. OpenAI's crawler page lists published IP files for OAI-SearchBot, GPTBot and ChatGPT-User and does not mention Web Bot Auth. We found no operator documentation saying that ClaudeBot, Claude-User or the main Perplexity crawlers sign requests. Treat them as unsigned and keep verifying them by IP list, as described in the verification guide above. Check each operator's page again, because this list changes.
How to check whether a request is signed
Your server sees a signature only if the headers reach it. A CDN or proxy in front of your origin may consume or strip them, so an empty log does not prove nothing was signed.
At an nginx origin you can log them with the $http_ prefix, which turns a header name into a variable:
log_format signed '$remote_addr "$request" $status '
'agent=$http_signature_agent input=$http_signature_input';
access_log /var/log/nginx/signed.log signed;Then look for rows where the agent field is not empty:
grep -v 'agent=-' /var/log/nginx/signed.log | tail -20Remember that a logged header is only a claim. Anyone can send Signature-Agent: "https://chatgpt.com". Only a verification step, at your CDN or in your own code, turns it into evidence. On Cloudflare, check the security events for the verified bot classification. On AWS WAF, look for the web_bot_auth labels in your logs. If you are a bot operator, Cloudflare documents a test endpoint at crawltest.com/cdn-cgi/web-bot-auth that returns 200 for a verified message, 401 for an unknown key or failed verification, and 400 for other errors.
What to do when a request is not signed
Most requests will not be signed, so an unsigned request is not evidence of a fake. Your options, in order:
- Fall back to the older checks. Compare the source address with the operator's published IP list or confirm reverse DNS, as in verifying real AI crawlers.
- Allow by signature where you can, not only by it. An allow rule keyed to a verified signature helps agents that sign. A rule that denies everything unsigned would also deny crawlers that never sign, including ones you may want.
- Check challenge behavior. If your WAF serves a challenge page to a crawler that does not sign, the crawler reads the challenge instead of your content. WAF challenge pages and AI crawlers explains how to detect that.
Status of the standard and limits of relying on it
Web Bot Auth is not finished. The current Internet-Draft is draft-ietf-webbotauth-httpsig-protocol, an active working group draft at version 01 that the Datatracker showed as updated on 29 October 2026 and expiring on 2 May 2027. It replaces the earlier draft-meunier-webbotauth-httpsig-protocol, which the Datatracker now marks as expired and replaced. Neither is an RFC. Google, AWS and Akamai all describe their support as based on a draft or an emerging standard.
Some limits follow from that:
- Details can change. The
Signature-Agentformat already differs between sources. - Coverage is partial. It works only where both the operator signs and your platform verifies.
- It proves origin, not intent. Akamai says Web Bot Auth verifies origin rather than defining trust. A signed bot can still be rate limited or blocked by you.
- Platform support varies. AWS's AgentCore documentation says not every site will recognise signed traffic, and that site owners keep full control over their policies.
Frequently asked questions
Does Web Bot Auth replace robots.txt?
No. It answers who sent a request, while robots.txt tells a crawler what you want it to fetch. A verified bot can still be disallowed in robots.txt, and Google notes that its user-triggered fetchers generally do not follow robots.txt anyway.
Is Web Bot Auth an official standard?
Not yet. It builds on RFC 9421, which is a Proposed Standard, but the Web Bot Auth protocol itself is an IETF Internet-Draft at the time of writing.
Do GPTBot and OAI-SearchBot sign their requests?
OpenAI's crawler documentation does not say so. It lists published IP ranges for OAI-SearchBot, GPTBot and ChatGPT-User. OpenAI does document signing for ChatGPT agent.
Do I need to set anything up to benefit from it?
Not if you use a CDN that verifies signatures: Cloudflare, Akamai and AWS WAF Bot Control describe doing it for you. You only need to decide what policy applies to verified traffic, and keep that policy from blocking agents you want.
Check what a crawler sees on your page
Signature checks decide whether a bot reaches your server, but the page still has to be readable once it does. The free check reads one page the way AI crawlers do, with no account needed. It runs from Crawlito's servers, so your CDN's security log remains the evidence for what it does to real crawlers. If you want the whole site reviewed, an audit covers up to 500 pages with evidence and prioritized fixes.
Filed under CDN and bot protection. Topics: bot-protection, ai-crawlers, waf, cloudflare, http-status