Skip to content

Which WAF blocked the crawler? Read the headers and block page

Find which WAF or CDN is blocking a crawler from its response headers, status code and block page text, then send the owner the request ID they need to fix it.

By Crawlito · 8 min read

In short

  • Most CDNs and WAFs stamp their own header on every response: cf-ray for Cloudflare, X-Azure-Ref for Azure Front Door, X-Served-By for Fastly, x-vercel-id for Vercel and Via ending in cloudfront.net for CloudFront.
  • Default block statuses differ by vendor: 403 for Cloudflare WAF blocks, AWS WAF and Azure Front Door WAF, 429 for Cloudflare and Front Door rate limits, and a 406 block code for Fastly Next-Gen WAF.
  • Akamai and F5 are easier to name from the block page body, which carries an Akamai reference number or an F5 support ID.
  • When two vendors are stacked, headers from the inner one pass through, so the block marker and the ID in the body tell you which layer actually refused the request.
  • Send the site owner the URL, UTC time, user agent, status and the vendor's request ID, because that ID is how they find the matching log entry.
On this page

To find which WAF blocked a crawler, request the page with the crawler's user agent, save the response headers and body, and look for the vendor's own markers. Cloudflare adds cf-ray, Azure Front Door adds X-Azure-Ref, Fastly adds X-Served-By, Vercel adds x-vercel-id and CloudFront sets a Via header ending in cloudfront.net (CloudFront). The status code and the text of the block page usually confirm it: Fastly's Next-Gen WAF uses 406 as its default block code, Akamai's page shows a reference number and F5's shows a support ID.

This matters most when you audit a site you do not run. The fix lives in one specific console, and "your firewall blocks GPTBot" sent to the wrong team goes nowhere. If you need the background on why a block or challenge hides your content, read why a WAF challenge page shows AI bots nothing first. This post is the lookup table.

How to capture the evidence with curl

You need three things from one request: the status, the headers and the body. Save this as which-waf.sh and run bash which-waf.sh https://example.com/page. It does not follow redirects, so you see who answered the first hop; if you get a 3xx, run it again on the location URL.

bash
url="${1:?give a URL}"
ua="${2:-Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot}"
curl -sS -o body.html -D headers.txt -A "$ua" -H 'Accept: text/html' --max-time 20 "$url"
echo "status: $(awk 'toupper($1) ~ /^HTTP\// {c=$2} END {print c}' headers.txt)"
grep -iE '^(server|via|cf-ray|cf-mitigated|x-cache|x-amz-cf-id|x-amz-cf-pop|x-amzn-waf-action|x-azure-ref|x-served-by|x-vercel-id|x-vercel-mitigated|x-sucuri-id):' headers.txt
grep -ioE 'ray id|reference #[0-9a-f.]+|support id|request blocked|the request is blocked|access denied' body.html | sort -u

Run it twice: once with the crawler's user agent and once with a browser string as the second argument. Two caveats from the wider curl testing guide apply here too. Your request comes from your IP, not the operator's, so an IP or verified-bot rule can treat you differently from the real crawler. And a single request will not trip a rate limit that the real crawler hits. If your test passes but the site owner's logs show blocks, the headers in those log entries are your evidence instead.

Which response headers name the CDN or WAF

These are the headers each vendor documents, or in one case that is widely seen but not documented. A header tells you the request passed through that network, not that it was the layer that blocked it. The next sections deal with that.

HeaderExample valueVendorWhat the vendor says
cf-ray230b030023ae2822-SJCCloudflareA hashed Ray ID that ends in a three-letter code for the data center (Cloudflare HTTP headers)
cf-mitigatedchallengeCloudflareSet to challenge on every challenge page type (Cloudflare challenge detection)
Via1.1 1026589cc7887e7a0dc7827b4example.cloudfront.net (CloudFront)Amazon CloudFrontCloudFront sets this value on responses to viewers (CloudFront custom origin behavior)
x-amz-cf-idAn opaque request IDAmazon CloudFrontThe request ID, also logged as x-edge-request-id and as requestId in AWS WAF logs (AWS edge troubleshooting guide)
X-CacheError from cloudfrontAmazon CloudFrontSet on errors CloudFront generated itself, such as its 502 responses (CloudFront 502 guide)
x-amzn-waf-actionchallenge or captchaAWS WAFMarks a Challenge (202) or CAPTCHA (405) response (AWS WAF CAPTCHA and Challenge)
X-Azure-RefA long base64-like stringAzure Front DoorAttached to all responses; used to search access logs (Front Door headers)
X-Served-Bycache-{datacenter}{nodeid}-{datacenter}FastlySet by default on all responses Fastly processes, naming the delivery node; shielding or Next-Gen WAF at Edge can add more comma-separated entries (Fastly reference)
x-vercel-idRegion codesVercelThe Vercel regions the request hit (Vercel response headers)
serverVercelVercelVercel notes another proxy, such as Cloudflare, can override it
x-vercel-mitigateddeny or challengeVercelNot in Vercel's response header reference at the time of writing; seen on 403 responses in Vercel Community reports (deny, challenge)
x-sucuri-id14002SucuriThe firewall node that answered (Sucuri cache headers)

Fastly also warns that node IDs and data center codes can be reused, so an X-Served-By value only describes that moment. Cloudflare says Ray IDs are not guaranteed to be unique, and because Security Events are sampled, a missing entry does not prove Cloudflare took no action (Cloudflare Ray ID).

Status codes and block pages by vendor

The status code narrows the list, and the body usually settles it. Defaults can be changed by the site owner, so treat this table as the starting point, not proof.

VendorDefault block statusWhat the body showsID to quote
Cloudflare WAF, IP or country block403A Cloudflare-branded error pagecf-ray value
Cloudflare rate limiting429Rate limiting pagecf-ray value
Cloudflare managed challenge403Challenge HTML, plus cf-mitigated: challengecf-ray value
AWS WAF on CloudFront403CloudFront's error page, or the owner's custom responsex-amz-cf-id value
Azure Front Door WAF403, 429 for rate limits"The request is blocked." with a tracking referenceX-Azure-Ref value
Fastly Next-Gen WAF406 block code, changeableVariesTime, IP and path
Vercel firewall deny403Varies; challenges show a checkpoint pagex-vercel-id value
AkamaiNot documented"Access Denied" with "Reference #"The reference number
F5 BIG-IP Advanced WAFVariesA rejection message with a support IDThe support ID

Cloudflare

Cloudflare's error page types list a 403 for WAF blocks, IP and country blocks and managed challenges, and a 429 for rate limiting blocks. On Pro plans and above, a custom rule's Block action can send a custom response with any status from 400 to 499, so a 403 is not guaranteed. If the body carries a four-digit code such as 1010 or 1020, look it up in Cloudflare errors 1010 and 1020 for crawlers.

Amazon CloudFront and AWS WAF

AWS WAF's default Block response is 403 (Forbidden), according to its custom response documentation, and the owner can replace the status, headers and body. CloudFront's 403 troubleshooting page adds a warning that matters here: CloudFront cannot tell a 403 from the origin apart from a 403 that AWS WAF returned, so the owner has to check the web ACL's rules for a blocked request. A 202 or 405 with x-amzn-waf-action is a challenge or CAPTCHA, not a block.

Azure Front Door

Microsoft's custom response page says a Front Door WAF block returns 403 with "The request is blocked." and a tracking reference that links to the logs. An empty custom body gives 403 for normal blocks and 429 for rate limit blocks.

Fastly

Fastly documents 406 as the Next-Gen WAF's default agent response code for blocked requests, and the owner can change it. Fastly also notes that the status the client finally receives comes from the module or proxy that handles the request, so a 406 to a crawler that sent an ordinary Accept: text/html is a strong hint, not proof. The Next-Gen WAF can also be deployed on the site's own servers, so a 406 without X-Served-By can still be Fastly's product.

Vercel

A Vercel deny returns 403 Forbidden and the request never reaches the application, according to Vercel's firewall concepts. Challenges show the Vercel Security Checkpoint; Vercel Security Checkpoint and crawlers covers that page. Because x-vercel-mitigated is not documented, treat it as a strong clue, and use x-vercel-id as the reference.

Akamai and F5

Neither has a single documented response header to rely on, so read the body. Microsoft documents the Akamai page its own Akamai-hosted sites show as "Access Denied" with a reference number that is a unique identifier for that access attempt (Microsoft Learn); Akamai Access Denied reference numbers explains what to do with it. F5's blocking response manual says every default response page carries a support ID that the system fills in when it issues the page. That manual does not quote the default wording; F5 BIG-IP requested URL rejected covers the page itself.

When the headers point at two vendors

Sites often stack layers: Cloudflare in front of Vercel, CloudFront in front of a server with its own WAF, Front Door in front of an app with a security plugin. Front Door passes the backend's headers through to the client, and Vercel says its server header can be overridden by Cloudflare. So a 403 carrying both cf-ray and x-vercel-id tells you both networks were involved, not which one refused.

Use these tie-breakers:

  1. Match the block page to the marker. A Cloudflare-branded error page was written by Cloudflare. A plain page with "The request is blocked." was written by Front Door, whatever else is in the headers.
  2. Read CloudFront's X-Cache, but do not lean on it. AWS documents X-Cache: Error from cloudfront for errors CloudFront generated itself, such as its 502 responses. AWS does not document the value for WAF blocks, and it says CloudFront cannot tell an origin 403 from an AWS WAF 403, so the web ACL logs settle it.
  3. Ask Front Door for the origin status. Send X-Azure-DebugInfo: 1 and Front Door adds X-Azure-OriginStatusCode, the status the backend returned, which Microsoft says can differ from the status the client gets. If the origin status is also 403, the backend refused; if it differs or is missing, check Front Door's WAF log.
  4. Look at the branding. If the 403 body is the site's own styled error page, the origin or a plugin answered, not the CDN.

The Front Door check, with the same user agent as the script above:

bash
url="https://example.com/page"
ua="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
curl -s -o /dev/null -D - -A "$ua" -H 'X-Azure-DebugInfo: 1' "$url" | grep -i '^x-azure'

What to send the site owner

The vendor's request ID is what turns a complaint into a log lookup. Collect these for each failing request:

URL:         https://example.com/pricing
Time (UTC):  2026-11-15 09:42:10
User agent:  GPTBot/1.4 (full string as sent)
Source IP:   the IP the test ran from
Status:      403
Vendor:      Cloudflare (cf-ray present, Cloudflare block page)
Request ID:  cf-ray 230b030023ae2822-SJC

Then point them at the right place: Cloudflare Security Events (filter by Ray ID), AWS WAF logs for the web ACL (the requestId field), Front Door's access and WAF logs (the TrackingReference field matches X-Azure-Ref), Fastly's Next-Gen WAF console, the Firewall section of the Vercel dashboard, Akamai support with the reference number, or the BIG-IP administrator with the support ID. Ask them to allow the crawler by verified identity rather than user agent alone, as the challenge page guide explains.

Frequently asked questions

Does a cf-ray header mean Cloudflare blocked the request?

No. It means the request passed through Cloudflare. Cloudflare blocked it if the body is a Cloudflare error or challenge page, or if cf-mitigated: challenge is present.

Is a 406 always Fastly?

No, but it is the default block code of Fastly's Next-Gen WAF, so it is the first product to check. Look for X-Served-By and ask the owner whether they run that WAF.

Why does my test pass when the site owner's logs show the crawler blocked?

Your request comes from your IP, while rules on IP reputation, country or verified-bot status apply to the crawler's real addresses. Use the headers and IDs recorded in the owner's logs as the evidence instead.

Can the site owner hide these headers?

Some, yes: block pages and statuses can be customized, and a proxy in front can rewrite server. Request IDs such as X-Azure-Ref and cf-ray are what each vendor's logs are searched by, so they are the markers worth recording.

Check what the crawler gets

If a page you care about returns a block, start with the free check, which fetches one public page as AI crawlers do and needs no account. For a client site with several layers in front of it, an audit checks up to 500 pages and gives the owner a PDF with the evidence and prioritized fixes.

Filed under CDN and bot protection. Topics: waf, bot-protection, curl, cdn, agencies, troubleshooting