Fastly bot management and AI crawlers: verified bot signals
How Fastly bot management labels AI crawlers with VERIFIED-BOT and SUSPECTED-BOT signals, why blocked crawlers see 406, and how to write rules on signals.
By Crawlito · 9 min read
In short
- Fastly's Next-Gen WAF tags AI traffic with four system signals: VERIFIED-BOT.AI-CRAWLER, VERIFIED-BOT.AI-FETCHER, SUSPECTED-BOT.AI-CRAWLER and SUSPECTED-BOT.AI-FETCHER, added in April 2025.
- Fastly says it adds a VERIFIED-BOT signal only when the connecting address belongs to the vendor the bot names; it does not publish the criteria behind the SUSPECTED-BOT signals.
- Requests blocked by the Next-Gen WAF get a 406 agent response code by default, which you can change to 301, 302 or any code from 400 to 599.
- Fastly describes AI crawlers as bots for building AI models or indexes, so a block on VERIFIED-BOT.AI-CRAWLER can also stop crawlers that feed AI search.
- In Next-Gen WAF rules an Allow action always beats a Block, and Fastly recommends signal exclusion rules over Allow rules for trusted automation because Allow skips all WAF inspection.
On this page
Fastly bot management handles AI crawlers through Next-Gen WAF system signals, not through a fixed AI block. Each request it analyzes can carry a signal such as VERIFIED-BOT.AI-CRAWLER or SUSPECTED-BOT.AI-FETCHER, and your request rules decide what happens to it. To keep AI search crawlers and search engines while stopping scrapers, write rules on the verified and suspected signals rather than on user agent strings, and remember that anything you block gets a 406 by default.
This guide covers what the signals mean according to Fastly's documentation, what is not documented, how to build rules on them, and how to check what a crawler received. If you also run Akamai or AWS, the Akamai Bot Manager guide and the AWS WAF Bot Control guide cover the equivalent settings there.
What the Fastly bot signals are
Fastly's system signals reference says bot signals are "labels that describe suspected and verified bot requests". Fastly added the four AI signals in April 2025, according to its changelog. The ones that matter for AI search readiness:
| Signal | Fastly's description, in short | Usable in |
|---|---|---|
VERIFIED-BOT.AI-CRAWLER | Crawlers used for training AIs and LLMs, generally for building AI models or indexes | Lists, rate limit rules, request rules, signal exclusions |
VERIFIED-BOT.AI-FETCHER | Fetchers used by AIs and LLMs to enrich results in response to a user query | Lists, rate limit rules, request rules, signal exclusions |
SUSPECTED-BOT.AI-CRAWLER | A request suspected of being an AI or LLM crawler | Lists, rate limit rules, request rules, signal exclusions |
SUSPECTED-BOT.AI-FETCHER | A request suspected of being an AI fetcher for user queries | Lists, rate limit rules, request rules, signal exclusions |
VERIFIED-BOT.SEARCH-ENGINE-CRAWLER | Crawlers that index your site for search engines | Lists, rate limit rules, request rules, signal exclusions |
SUSPECTED-BOT | A request suspected of being a bot | Rate limit rules, request rules, signal exclusions |
SUSPECTED-BAD-BOT | A request suspected of being a bad bot | Lists, rate limit rules, request rules, signal exclusions |
IMPOSTOR | Someone pretending to be a Google or Bing search bot | Templated rules only |
There are other verified categories too, including VERIFIED-BOT.CONTENT-FETCHER, VERIFIED-BOT.PAGE-PREVIEW and VERIFIED-BOT.SEARCH-ENGINE-OPTIMIZATION. IMPOSTOR sits in the anomaly signals rather than the bot signals, and Fastly notes that it should not be used as an indicator of malicious intent.
Two availability limits apply. Fastly says bot signals are only available to Edge WAF deployments unless a signal's description says otherwise, and SUSPECTED-BOT and SUSPECTED-BAD-BOT also work on On-Prem WAF. Fastly's product page for Bot Management says it must be bought with Full-Site Delivery or Streaming Delivery and is not available on the Essentials platform.
Crawler versus fetcher is not training versus search
The training, search and user-triggered split does not map neatly onto Fastly's two AI categories. AI-FETCHER is close to the user-triggered group, such as ChatGPT-User, Claude-User and Perplexity-User. AI-CRAWLER is described as both training crawlers and bots that build "AI models or indexes", and an index is what AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build.
Fastly does not publish which named bot falls under which signal. Treat a block on VERIFIED-BOT.AI-CRAWLER as one that may stop AI search crawlers along with training crawlers, and check your own request data before you rely on the category.
What verified and suspected mean
Fastly's October 2026 research post on AI bot impostors explains the verified side. Fastly does not take the user agent at its word: it checks whether the connecting address belongs to the vendor the bot names, and attaches a VERIFIED-BOT signal when it does. The post notes that AI vendors usually make their bots verifiable by publishing IP ranges.
The suspected side is less documented. The signals reference only says a request "is suspected of being" a bot of that type, with no criteria. In practice, read a suspected AI signal as "looks like or claims to be an AI bot, and was not verified". That covers scrapers copying a crawler's user agent, and it can also cover a real crawler arriving from an address Fastly does not recognize. The same research post counted about 6.3 million requests from nearly 10,200 IP addresses in one day (23 September 2026) that claimed a verifiable AI bot identity and failed verification, which is the case a suspected-signal rule is for.
The practical consequence for testing: a curl request from your laptop with OAI-SearchBot's user agent is not the verified OAI-SearchBot. It will not carry a verified signal, and it may carry a suspected one. Your result tells you how impostors are treated, not how the real crawler is. The guide to verifying real AI crawlers shows how to confirm a crawler's identity from the operator's published IP ranges.
Why blocked crawlers get a 406
Fastly's agent response code documentation says that by default, blocked requests receive a 406 agent response code. You can change the blocking response code for a workspace: supported values are 301, 302 and 400 to 599. Fastly also notes that the HTTP status the client receives can differ from the agent response code because of internal business logic.
A 406 Not Acceptable normally means content negotiation failed, so it is easy to misread. If a crawler shows 406 responses in your edge logs and your origin never content-negotiates, suspect a Next-Gen WAF block first. A crawler that gets one does not receive your page, whichever code you choose. The planned post on which WAF blocked a crawler covers telling vendors apart from response details.
Some agent response codes are not blocks. Fastly's troubleshooting page says -2, -1 and 0 mark requests that were not processed correctly, and that on a module timeout, typically 100ms, the module fails open and lets the request through.
How to build rules on signals, not user agents
Fastly's rules overview lists the actions you can use: Add signal, Allow, Block, Browser challenge, Dynamic challenge, Deception, Exclude signal, Log and Verify token, with support varying by rule type. Two precedence rules shape everything:
- A rule with an Allow action always takes precedence over a rule with a Block action.
- An account (corp) rule usually takes precedence over a workspace (site) rule, except when the workspace rule has an Allow action.
Conditions use the fields in Fastly's rule conditions page, including Signal, User agent, IP address, Path and JA3 Fingerprint. Signal is a multi-value field with a type property, and multi-value fields use the Exists where and Does not exist where operators with subconditions. The rules below are plain-text summaries of the logic, not the console's exact labels.
Rule 1: block unverified AI bots
This stops scrapers posing as AI crawlers without touching verified ones:
Rule type: Request rule
Conditions: Any of
Signal exists where Signal type equals SUSPECTED-BOT.AI-CRAWLER
Signal exists where Signal type equals SUSPECTED-BOT.AI-FETCHER
Action: BlockStart with Add signal or a narrow path scope if you are unsure, review the tagged requests, then switch to Block.
Rule 2: decide on training crawlers separately
If you want to opt out of AI training at the edge, do not block the whole VERIFIED-BOT.AI-CRAWLER category, for the reason above. Combine the verified signal with the specific training crawlers you mean:
Rule type: Request rule
Conditions: All of
Signal exists where Signal type equals VERIFIED-BOT.AI-CRAWLER
User agent matches (regexp) (?i)(GPTBot|ClaudeBot|CCBot)
Action: BlockHere the user agent only picks which verified bot you mean; verification still comes from the signal, so a scraper copying GPTBot does not match this rule, and if Fastly tags it as a suspected AI bot, Rule 1 handles it. A robots.txt Disallow for the same crawlers is the cleaner policy statement, and those operators document that they respect it. The Matches and Is in list operators are not included with the Essential platform.
Do not challenge crawlers
Fastly says a Browser challenge sends either a non-interactive JavaScript proof-of-work or an interactive CAPTCHA, and Dynamic challenge can pick between Private Access Tokens, proof-of-work and CAPTCHA. Crawlers generally do not solve these, so a challenge on AI search crawlers acts like a block. See WAF challenge pages and AI crawlers.
Prefer signal exclusions to Allow rules
If a bot signal catches your own monitoring or a partner integration, Fastly's post on stopping bad bots recommends a signal exclusion rule instead of an Allow rule. An exclusion only stops a signal being tagged, while an Allow means the request bypasses all WAF inspection. The same applies to crawlers: a broad Allow on a verified signal also exempts those requests from your attack rules.
Bot Management before and after the cache
Fastly's Bot Management overview describes two deployments:
| Mode | What it inspects | Requires |
|---|---|---|
| Post-cache inspection | Cache-miss requests, alongside the Next-Gen WAF | Bot Management and Next-Gen WAF, Edge WAF deployment only |
| Pre-cache inspection (ContentGuard) | All requests, before they reach the cache | Bot Management |
With post-cache inspection, Bot Management only analyzes cache-miss requests, so a crawler request served from cache is not analyzed there. That is usually fine for AI search access, but it means the bot data you see can undercount crawler visits.
ContentGuard exposes bot results to VCL. Fastly's ContentGuard guide documents variables such as fastly.bot.analyzed, fastly.bot.detected, fastly.bot.name, fastly.bot.category.is_verified, fastly.bot.category.is_ai_crawler, fastly.bot.category.is_ai_fetcher and fastly.bot.category.is_search_engine_crawler. Its sample code returns error 402 "AI Licensing Required" for every AI crawler and AI fetcher. Copied unchanged, that shuts out AI search crawlers and user-triggered fetchers too. A narrower version that only refuses AI bots Fastly could not verify:
sub vcl_recv {
if (fastly.bot.analyzed && fastly.bot.detected && !fastly.bot.category.is_verified) {
if (fastly.bot.category.is_ai_crawler || fastly.bot.category.is_ai_fetcher) {
error 403 "Forbidden";
}
}
}Fastly's documentation does not say whether the AI category variables are set for unverified bots, so test this on your service before relying on it.
How to check what a crawler received
- Open the request details. Fastly says you can view the agent response code for an individual request on its request details page.
- Look at the signals. Look for real crawlers tagged
VERIFIED-BOT.AI-CRAWLERorVERIFIED-BOT.AI-FETCHERwith a blocking response. Those are the requests your rules should not be stopping. - Check the origin logs. A drop in hits from OAI-SearchBot or PerplexityBot at your origin, with no change at the edge, points to a block or a cache effect. Filter on the crawler user agents and compare status codes over time.
- Recheck after each change. Confirm verified crawlers now get
200responses and the impostor traffic is still blocked.
Frequently asked questions
Does Fastly block AI crawlers by default?
Fastly's documentation does not describe a default block on AI signals. The signals label requests, and your request rules decide whether to block them, so check your workspace and corp rules for anything acting on the AI signals.
What is the difference between VERIFIED-BOT.AI-CRAWLER and VERIFIED-BOT.AI-FETCHER?
Fastly describes AI crawlers as bots used for building AI models or indexes, and AI fetchers as bots that enrich results in response to a user query. Fastly does not publish which named bot falls under each.
Why do crawlers get a 406 from my Fastly site?
By default the Next-Gen WAF returns a 406 agent response code for blocked requests. Check the agent response code on the request's details page, or change the workspace's blocking code to make WAF blocks easier to spot.
Can I allow verified AI crawlers with an Allow rule?
You can, and Allow beats any Block, but an Allow rule skips all WAF inspection for those requests. Fastly recommends signal exclusion rules for trusted automation, and blocking only the suspected signals keeps verified crawlers working without a broad Allow.
Check what your edge returns to AI crawlers
After changing Fastly rules, run the free check on an important page to see what the main AI crawler user agents get back. It runs from Crawlito's servers, not from the crawlers' own addresses, so it tests the unverified path, which is still the quickest way to spot a rule blocking by user agent. For a review of edge rules, robots.txt and crawler evidence across a whole site, an audit covers up to 500 pages with prioritized fixes.
Filed under CDN and bot protection. Topics: bot-protection, waf, cdn, ai-crawlers