Which WAF blocked the crawler? Read the headers and block page
Find which WAF or CDN is blocking a crawler from its response headers, status code and block page text, then send the owner the request ID they need to fix it.
By Crawlito · 8 min read
In short
- Most CDNs and WAFs stamp their own header on every response: cf-ray for Cloudflare, X-Azure-Ref for Azure Front Door, X-Served-By for Fastly, x-vercel-id for Vercel and Via ending in cloudfront.net for CloudFront.
- Default block statuses differ by vendor: 403 for Cloudflare WAF blocks, AWS WAF and Azure Front Door WAF, 429 for Cloudflare and Front Door rate limits, and a 406 block code for Fastly Next-Gen WAF.
- Akamai and F5 are easier to name from the block page body, which carries an Akamai reference number or an F5 support ID.
- When two vendors are stacked, headers from the inner one pass through, so the block marker and the ID in the body tell you which layer actually refused the request.
- Send the site owner the URL, UTC time, user agent, status and the vendor's request ID, because that ID is how they find the matching log entry.
On this page
To find which WAF blocked a crawler, request the page with the crawler's user agent, save the response headers and body, and look for the vendor's own markers. Cloudflare adds cf-ray, Azure Front Door adds X-Azure-Ref, Fastly adds X-Served-By, Vercel adds x-vercel-id and CloudFront sets a Via header ending in cloudfront.net (CloudFront). The status code and the text of the block page usually confirm it: Fastly's Next-Gen WAF uses 406 as its default block code, Akamai's page shows a reference number and F5's shows a support ID.
This matters most when you audit a site you do not run. The fix lives in one specific console, and "your firewall blocks GPTBot" sent to the wrong team goes nowhere. If you need the background on why a block or challenge hides your content, read why a WAF challenge page shows AI bots nothing first. This post is the lookup table.
How to capture the evidence with curl
You need three things from one request: the status, the headers and the body. Save this as which-waf.sh and run bash which-waf.sh https://example.com/page. It does not follow redirects, so you see who answered the first hop; if you get a 3xx, run it again on the location URL.
url="${1:?give a URL}"
ua="${2:-Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot}"
curl -sS -o body.html -D headers.txt -A "$ua" -H 'Accept: text/html' --max-time 20 "$url"
echo "status: $(awk 'toupper($1) ~ /^HTTP\// {c=$2} END {print c}' headers.txt)"
grep -iE '^(server|via|cf-ray|cf-mitigated|x-cache|x-amz-cf-id|x-amz-cf-pop|x-amzn-waf-action|x-azure-ref|x-served-by|x-vercel-id|x-vercel-mitigated|x-sucuri-id):' headers.txt
grep -ioE 'ray id|reference #[0-9a-f.]+|support id|request blocked|the request is blocked|access denied' body.html | sort -uRun it twice: once with the crawler's user agent and once with a browser string as the second argument. Two caveats from the wider curl testing guide apply here too. Your request comes from your IP, not the operator's, so an IP or verified-bot rule can treat you differently from the real crawler. And a single request will not trip a rate limit that the real crawler hits. If your test passes but the site owner's logs show blocks, the headers in those log entries are your evidence instead.
Which response headers name the CDN or WAF
These are the headers each vendor documents, or in one case that is widely seen but not documented. A header tells you the request passed through that network, not that it was the layer that blocked it. The next sections deal with that.
| Header | Example value | Vendor | What the vendor says |
|---|---|---|---|
cf-ray | 230b030023ae2822-SJC | Cloudflare | A hashed Ray ID that ends in a three-letter code for the data center (Cloudflare HTTP headers) |
cf-mitigated | challenge | Cloudflare | Set to challenge on every challenge page type (Cloudflare challenge detection) |
Via | 1.1 1026589cc7887e7a0dc7827b4example.cloudfront.net (CloudFront) | Amazon CloudFront | CloudFront sets this value on responses to viewers (CloudFront custom origin behavior) |
x-amz-cf-id | An opaque request ID | Amazon CloudFront | The request ID, also logged as x-edge-request-id and as requestId in AWS WAF logs (AWS edge troubleshooting guide) |
X-Cache | Error from cloudfront | Amazon CloudFront | Set on errors CloudFront generated itself, such as its 502 responses (CloudFront 502 guide) |
x-amzn-waf-action | challenge or captcha | AWS WAF | Marks a Challenge (202) or CAPTCHA (405) response (AWS WAF CAPTCHA and Challenge) |
X-Azure-Ref | A long base64-like string | Azure Front Door | Attached to all responses; used to search access logs (Front Door headers) |
X-Served-By | cache-{datacenter}{nodeid}-{datacenter} | Fastly | Set by default on all responses Fastly processes, naming the delivery node; shielding or Next-Gen WAF at Edge can add more comma-separated entries (Fastly reference) |
x-vercel-id | Region codes | Vercel | The Vercel regions the request hit (Vercel response headers) |
server | Vercel | Vercel | Vercel notes another proxy, such as Cloudflare, can override it |
x-vercel-mitigated | deny or challenge | Vercel | Not in Vercel's response header reference at the time of writing; seen on 403 responses in Vercel Community reports (deny, challenge) |
x-sucuri-id | 14002 | Sucuri | The firewall node that answered (Sucuri cache headers) |
Fastly also warns that node IDs and data center codes can be reused, so an X-Served-By value only describes that moment. Cloudflare says Ray IDs are not guaranteed to be unique, and because Security Events are sampled, a missing entry does not prove Cloudflare took no action (Cloudflare Ray ID).
Status codes and block pages by vendor
The status code narrows the list, and the body usually settles it. Defaults can be changed by the site owner, so treat this table as the starting point, not proof.
| Vendor | Default block status | What the body shows | ID to quote |
|---|---|---|---|
| Cloudflare WAF, IP or country block | 403 | A Cloudflare-branded error page | cf-ray value |
| Cloudflare rate limiting | 429 | Rate limiting page | cf-ray value |
| Cloudflare managed challenge | 403 | Challenge HTML, plus cf-mitigated: challenge | cf-ray value |
| AWS WAF on CloudFront | 403 | CloudFront's error page, or the owner's custom response | x-amz-cf-id value |
| Azure Front Door WAF | 403, 429 for rate limits | "The request is blocked." with a tracking reference | X-Azure-Ref value |
| Fastly Next-Gen WAF | 406 block code, changeable | Varies | Time, IP and path |
| Vercel firewall deny | 403 | Varies; challenges show a checkpoint page | x-vercel-id value |
| Akamai | Not documented | "Access Denied" with "Reference #" | The reference number |
| F5 BIG-IP Advanced WAF | Varies | A rejection message with a support ID | The support ID |
Cloudflare
Cloudflare's error page types list a 403 for WAF blocks, IP and country blocks and managed challenges, and a 429 for rate limiting blocks. On Pro plans and above, a custom rule's Block action can send a custom response with any status from 400 to 499, so a 403 is not guaranteed. If the body carries a four-digit code such as 1010 or 1020, look it up in Cloudflare errors 1010 and 1020 for crawlers.
Amazon CloudFront and AWS WAF
AWS WAF's default Block response is 403 (Forbidden), according to its custom response documentation, and the owner can replace the status, headers and body. CloudFront's 403 troubleshooting page adds a warning that matters here: CloudFront cannot tell a 403 from the origin apart from a 403 that AWS WAF returned, so the owner has to check the web ACL's rules for a blocked request. A 202 or 405 with x-amzn-waf-action is a challenge or CAPTCHA, not a block.
Azure Front Door
Microsoft's custom response page says a Front Door WAF block returns 403 with "The request is blocked." and a tracking reference that links to the logs. An empty custom body gives 403 for normal blocks and 429 for rate limit blocks.
Fastly
Fastly documents 406 as the Next-Gen WAF's default agent response code for blocked requests, and the owner can change it. Fastly also notes that the status the client finally receives comes from the module or proxy that handles the request, so a 406 to a crawler that sent an ordinary Accept: text/html is a strong hint, not proof. The Next-Gen WAF can also be deployed on the site's own servers, so a 406 without X-Served-By can still be Fastly's product.
Vercel
A Vercel deny returns 403 Forbidden and the request never reaches the application, according to Vercel's firewall concepts. Challenges show the Vercel Security Checkpoint; Vercel Security Checkpoint and crawlers covers that page. Because x-vercel-mitigated is not documented, treat it as a strong clue, and use x-vercel-id as the reference.
Akamai and F5
Neither has a single documented response header to rely on, so read the body. Microsoft documents the Akamai page its own Akamai-hosted sites show as "Access Denied" with a reference number that is a unique identifier for that access attempt (Microsoft Learn); Akamai Access Denied reference numbers explains what to do with it. F5's blocking response manual says every default response page carries a support ID that the system fills in when it issues the page. That manual does not quote the default wording; F5 BIG-IP requested URL rejected covers the page itself.
When the headers point at two vendors
Sites often stack layers: Cloudflare in front of Vercel, CloudFront in front of a server with its own WAF, Front Door in front of an app with a security plugin. Front Door passes the backend's headers through to the client, and Vercel says its server header can be overridden by Cloudflare. So a 403 carrying both cf-ray and x-vercel-id tells you both networks were involved, not which one refused.
Use these tie-breakers:
- Match the block page to the marker. A Cloudflare-branded error page was written by Cloudflare. A plain page with "The request is blocked." was written by Front Door, whatever else is in the headers.
- Read CloudFront's
X-Cache, but do not lean on it. AWS documentsX-Cache: Error from cloudfrontfor errors CloudFront generated itself, such as its 502 responses. AWS does not document the value for WAF blocks, and it says CloudFront cannot tell an origin 403 from an AWS WAF 403, so the web ACL logs settle it. - Ask Front Door for the origin status. Send
X-Azure-DebugInfo: 1and Front Door addsX-Azure-OriginStatusCode, the status the backend returned, which Microsoft says can differ from the status the client gets. If the origin status is also 403, the backend refused; if it differs or is missing, check Front Door's WAF log. - Look at the branding. If the 403 body is the site's own styled error page, the origin or a plugin answered, not the CDN.
The Front Door check, with the same user agent as the script above:
url="https://example.com/page"
ua="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
curl -s -o /dev/null -D - -A "$ua" -H 'X-Azure-DebugInfo: 1' "$url" | grep -i '^x-azure'What to send the site owner
The vendor's request ID is what turns a complaint into a log lookup. Collect these for each failing request:
URL: https://example.com/pricing
Time (UTC): 2026-11-15 09:42:10
User agent: GPTBot/1.4 (full string as sent)
Source IP: the IP the test ran from
Status: 403
Vendor: Cloudflare (cf-ray present, Cloudflare block page)
Request ID: cf-ray 230b030023ae2822-SJCThen point them at the right place: Cloudflare Security Events (filter by Ray ID), AWS WAF logs for the web ACL (the requestId field), Front Door's access and WAF logs (the TrackingReference field matches X-Azure-Ref), Fastly's Next-Gen WAF console, the Firewall section of the Vercel dashboard, Akamai support with the reference number, or the BIG-IP administrator with the support ID. Ask them to allow the crawler by verified identity rather than user agent alone, as the challenge page guide explains.
Frequently asked questions
Does a cf-ray header mean Cloudflare blocked the request?
No. It means the request passed through Cloudflare. Cloudflare blocked it if the body is a Cloudflare error or challenge page, or if cf-mitigated: challenge is present.
Is a 406 always Fastly?
No, but it is the default block code of Fastly's Next-Gen WAF, so it is the first product to check. Look for X-Served-By and ask the owner whether they run that WAF.
Why does my test pass when the site owner's logs show the crawler blocked?
Your request comes from your IP, while rules on IP reputation, country or verified-bot status apply to the crawler's real addresses. Use the headers and IDs recorded in the owner's logs as the evidence instead.
Can the site owner hide these headers?
Some, yes: block pages and statuses can be customized, and a proxy in front can rewrite server. Request IDs such as X-Azure-Ref and cf-ray are what each vendor's logs are searched by, so they are the markers worth recording.
Check what the crawler gets
If a page you care about returns a block, start with the free check, which fetches one public page as AI crawlers do and needs no account. For a client site with several layers in front of it, an audit checks up to 500 pages and gives the owner a PDF with the evidence and prioritized fixes.
Filed under CDN and bot protection. Topics: waf, bot-protection, curl, cdn, agencies, troubleshooting