Skip to content

AWS WAF Bot Control and AI crawlers: allow the verified ones

AWS WAF Bot Control blocks AI bots by default, verified or not. See how its labels and Web Bot Auth work and how to allow the AI crawlers you want.

By Crawlito · 8 min read

In short

  • In AWS's current documentation, the Bot Control CategoryAI rule blocks every AI bot it matches, verified or not, while the other category rules block only bots it cannot verify.
  • A custom Allow rule cannot rescue a request the Bot Control group has already blocked: override CategoryAI to Count first, then write label-match rules that run after the group.
  • Web Bot Auth verification needs a static rule group version of 4.0 or later, selected explicitly, and was limited to CloudFront until version 6.0 added regional resources.
  • Bot Control decides whether a bot is verified from the IP address of the request origin, so a real crawler that arrives through a proxy can look unverified.
  • The WAF log records the labels, the terminating rule and the action for each request, which is the evidence for whether a crawler was allowed.
On this page

AWS WAF Bot Control labels the AI crawlers it can identify, but its CategoryAI rule blocks AI bots whether they are verified or not. So if you run CloudFront or an Application Load Balancer with the Bot Control managed rule group, the crawlers you want may be getting a 403 even though robots.txt allows them. To let the right ones in, change that one rule to Count, then add label-based rules after the rule group that allow verified AI bots and block the rest. The details depend on your rule group version, so check yours first.

One caveat before the steps. Allowing a crawler removes a technical barrier. It does not mean an AI tool will use your pages in its answers.

How does Bot Control treat AI bots?

The managed rule group, AWSManagedRulesBotControlRuleSet, has two protection levels, according to AWS's rule group documentation. The Common level labels self-identifying bots and blocks the ones it cannot verify. The Targeted level adds detection for bots that do not identify themselves, using rate limiting, CAPTCHA and background browser challenges. You pay extra fees for using the rule group.

Two behaviors matter for AI traffic at the time of writing:

RuleActionApplies to
CategoryAIBlockAll matching AI bots, verified or unverified
CategorySearchEngine, CategoryContentFetcher and the other category rulesBlockOnly bots Bot Control cannot verify
SignalNonBrowserUserAgent, SignalKnownBotDataCenter, SignalAutomatedBrowserBlockOnly requests that are not from verified bots
TGT_ rules (Targeted level)Challenge, CAPTCHA, Block or Count, by ruleOnly requests that are not from verified bots

So a verified Googlebot passes the Common level without any rule of yours, while a verified AI crawler is blocked by CategoryAI unless something changes. AWS added CategoryAI to the rule group on 2023-08-30, according to its changelog.

Rule group versions change the behavior

Bot Control is versioned, and AWS documents only the most recent static version on the main page. The changelog entries that affect AI crawlers:

VersionDateChange that matters here
3.22025-05-29New AI bot labels, including bot:name:searchbot, bot:name:nova_act and bot:organization:openai
4.02025-11-20Web Bot Auth (WBA) support, CloudFront only. CategoryAI stops matching requests with the web_bot_auth:verified label. Not applied to the default version; you select it explicitly
5.02026-02-25More than 400 new bots, plus Page Preview and Webhooks categories. WBA still CloudFront only
6.02026-05-22WBA extended to regional commercial integrations. WBA-verified bots are treated as standard verified bots for all categories, not only AI

AWS says static versions do not change the default version's behavior. Open your web ACL and read which version the rule group uses before you trust any rule description, including the ones here. The changelog also lists later releases: 6.1 (2026-07-24) added new bot signatures, including AI ones, and 7.0 (2026-10-15) added token provenance and device ID labels.

Why do unverified AI bots get blocked or challenged?

Bot Control can recognize a request as a named bot, but recognizing it is not the same as verifying it. AWS says Bot Control uses the IP address of the web request origin to help decide whether a bot is verified, and that you cannot configure it to use the forwarded IP address.

A bot that identifies itself but publishes nothing that can be used to verify it independently gets the label awswaf:managed:aws:bot-control:bot:unverified. AWS says these bot signatures "can be falsified", so they are treated as unverified, and the category rules block unverified bots.

It also explains two problems for real crawlers:

  • A proxy in front of the WAF. If the web ACL sits behind another proxy or load balancer, the origin IP is the proxy's, and a real crawler can look unverified. AWS suggests adding a rule that runs before the Bot Control group, configured to use the forwarded IP and to explicitly allow the verified bots.
  • User-triggered bots. AWS describes a bot:user_triggered:verified label for bots similar to verified bots that end users may invoke directly, and says Bot Control treats them like unverified bots. AWS does not document which AI fetchers carry it, so look in your logs. The training, search and user-triggered split explains why those fetchers behave differently.

At the Targeted level, a client that cannot run JavaScript cannot complete a challenge. When the token is missing, invalid or expired, AWS WAF answers a Challenge action with HTTP 202 and a CAPTCHA action with 405, as covered in how challenge pages affect AI crawlers. A plain Block returns 403 by default.

What do the verified-bot and Web Bot Auth labels mean?

Rules add labels, and later rules in the same web ACL can match them. AWS says a label stays on the request until web ACL evaluation ends.

  • awswaf:managed:aws:bot-control:bot:verified marks a self-identifying bot that Bot Control could verify.
  • bot:name:<name>, bot:category:<category> and bot:organization:<organization> say which bot it is, such as bot:category:search_engine or bot:category:ai.
  • bot:unverified marks a bot that identifies itself, so it can be named and categorized, but does not publish information that can be used to verify its identity independently.
  • bot:web_bot_auth:verified, invalid, expired and unknown_bot report the result of a Web Bot Auth signature check.

Web Bot Auth is a cryptographic method. Operators publish public keys, sign their requests, and AWS WAF verifies the signature. Per AWS's security blog post (14 July 2026), agents on Amazon Bedrock AgentCore Browser sign automatically, while other agents must generate their own keys and sign requests. The sources I reviewed do not say which named AI search crawlers sign requests, so do not assume yours do. For the protocol itself, see Web Bot Auth and signed crawlers.

When web_bot_auth:verified is present, AWS says the category rules and TGT_TokenAbsent do not match, so the request proceeds.

How do you allow verified AI crawlers?

Rule order is the whole job. WAF evaluates rules by priority number, lowest first. A rule that matches a label must run after the rule that adds it, and Block and Allow end evaluation.

That is why AWS's own verified-bots example does not fix CategoryAI. Its Allow rule runs after the Bot Control group, and CategoryAI has already blocked the request by then.

Step 1: set CategoryAI to Count

Override the rule action inside the rule group. AWS documents this as RuleActionOverrides, and says Count counts the request and continues evaluating. Labels still reach later rules.

json
{
  "Name": "AWS-AWSManagedRulesBotControlRuleSet",
  "Priority": 10,
  "OverrideAction": { "None": {} },
  "Statement": {
    "ManagedRuleGroupStatement": {
      "VendorName": "AWS",
      "Name": "AWSManagedRulesBotControlRuleSet",
      "Version": "Version_7.0",
      "ManagedRuleGroupConfigs": [
        { "AWSManagedRulesBotControlRuleSet": { "InspectionLevel": "COMMON" } }
      ],
      "RuleActionOverrides": [
        { "Name": "CategoryAI", "ActionToUse": { "Count": {} } }
      ]
    }
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "BotControl"
  }
}

Use whichever static version you have chosen in place of Version_7.0.

Step 2: block AI bots that are not verified

Now CategoryAI no longer stops anything, fake AI bots included. Add a rule after the group that blocks AI-category requests unless they carry a verified label. I include the Web Bot Auth label because the sources do not say whether a WBA-verified request also gets bot:verified.

json
{
  "Name": "block-unverified-ai-bots",
  "Priority": 20,
  "Action": { "Block": {} },
  "Statement": {
    "AndStatement": {
      "Statements": [
        { "LabelMatchStatement": { "Scope": "LABEL", "Key": "awswaf:managed:aws:bot-control:bot:category:ai" } },
        { "NotStatement": { "Statement": { "OrStatement": { "Statements": [
          { "LabelMatchStatement": { "Scope": "LABEL", "Key": "awswaf:managed:aws:bot-control:bot:verified" } },
          { "LabelMatchStatement": { "Scope": "LABEL", "Key": "awswaf:managed:aws:bot-control:bot:web_bot_auth:verified" } }
        ] } } } }
      ]
    }
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "BlockUnverifiedAi"
  }
}

Step 3: allow verified bots your other rules block

If your own rules, such as geo or rate rules, block verified crawlers, add the AWS example rule: a label match on awswaf:managed:aws:bot-control:bot:verified with an Allow action, placed after Bot Control. Allow ends evaluation, so put any block rules for bots you do not want before it. Start with Count on new rules and check the logs before switching to Block. See geo-blocking and IP allowlists and rate limiting AI crawlers for the rules most likely to catch crawlers.

How do you block a category even when the bot is verified?

AWS says the rule group does not block the common verifiable bots it labels, apart from AI bots under CategoryAI. AWS says you can block them, or a subset, with a custom rule on the labels. An example from AWS's networking blog blocks a named bot, yandexbot, with a rule placed after the group. This version blocks a verified SEO crawler category:

json
{
  "Name": "block-seo-crawlers",
  "Priority": 15,
  "Action": { "Block": {} },
  "Statement": {
    "LabelMatchStatement": {
      "Scope": "LABEL",
      "Key": "awswaf:managed:aws:bot-control:bot:category:seo"
    }
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "BlockSeoBots"
  }
}

Swap in bot:name:bytespider, one of the AI bot name labels in the changelog, to block a single bot. To list every label for a version, call DescribeManagedRuleGroup; the labels are in AvailableLabels. Without a version name, AWS WAF uses the vendor's default version.

bash
aws wafv2 describe-managed-rule-group \
  --vendor-name AWS --name AWSManagedRulesBotControlRuleSet \
  --version-name Version_7.0 \
  --scope REGIONAL --region us-east-1 | jq -r '.AvailableLabels[].Name'

Use your own Region for regional web ACLs and the version your web ACL uses. For CloudFront web ACLs, use --scope CLOUDFRONT with --region us-east-1. Blocking by label is a policy choice: robots.txt only asks compliant bots to stay out.

How do you read the WAF logs for crawlers?

Turn on web ACL logging, then look for the crawler's user agent. AWS's log field reference gives the fields that matter:

  • action is the terminating action, and terminatingRuleId is the rule that ended evaluation. Default_Action means nothing terminated it.
  • ruleGroupList lists the rule groups that acted on the request, with match information. A matching rule you overrode to Count is listed as non-terminating, and AWS says its configured action is given in overriddenAction.
  • labels lists the labels on the request. AWS logs the first 100.
  • httpRequest.headers holds the User-Agent, and clientIp the origin IP.

On exported log files, this prints each AI-crawler request with its outcome and Bot Control labels:

bash
jq -r 'select(any(.httpRequest.headers[]?; (.name | ascii_downcase) == "user-agent"
  and (.value | test("OAI-SearchBot|GPTBot|ClaudeBot|PerplexityBot"))))
  | [.timestamp, .action, .terminatingRuleId, .httpRequest.clientIp,
     ((.labels // []) | map(.name | sub("awswaf:managed:aws:bot-control:"; "")) | join(","))]
  | @tsv' waf-logs.json

Read the result like this. A BLOCK from the Bot Control group with bot:category:ai means CategoryAI is still active. A block from your own rule with bot:unverified means the crawler failed verification, often for the proxy reason above. An ALLOW with bot:verified is the outcome you want. If a crawler request has no Bot Control labels, either Bot Control did not recognize it or an earlier rule ended evaluation first, so check rules with a lower priority number. For origin-side checks, see finding AI crawlers in server logs and verifying real AI crawlers.

Frequently asked questions

Does AWS WAF Bot Control block GPTBot and ClaudeBot by default?

In AWS's current documentation, CategoryAI has the Block action and applies to all matching AI bots, verified or not, when the rule group is added without overrides. AWS documents the label names but not the exact matching rules behind them, so confirm in your logs which labels a given crawler gets.

Will an Allow rule for the verified-bot label let verified AI crawlers through?

Not by itself. The Allow rule runs after the Bot Control group, and CategoryAI terminates evaluation first. Override CategoryAI to Count, then add your label-based rules.

Do I need Web Bot Auth to allow AI crawlers?

No. It is one way a request can be verified. It needs static version 4.0 or later, chosen explicitly, and it only helps for agents that sign their requests.

Why does my own curl test still get through or fail differently?

Verification is based on the origin IP, and your machine is not a crawler. A request with a crawler user agent from your IP is unverified, so it tests the unverified path. Use the logs for the real crawler's outcome.

Check what a crawler sees

The WAF log tells you what AWS decided for each crawler request. To see what a page returns to a client with no special treatment, the free check reads one page with no account needed. It runs from Crawlito's servers, so a WAF that verifies by IP can treat it differently from the real crawlers. For a whole site, the audit runs deterministic checks on up to 500 pages with evidence and prioritized fixes.

Filed under CDN and bot protection. Topics: bot-protection, waf, ai-crawlers, http-status, troubleshooting