Skip to content

Azure WAF Bot Manager rule set: how it classifies AI crawlers

The Azure WAF Bot Manager rule set has no AI rule. See which Bot ID an AI crawler matches, how to find it in WAF logs and how to change one rule's action.

By Crawlito · 9 min read

In short

  • Microsoft's Bot Manager 1.0 and 1.1 rule tables list BadBots, GoodBots and UnknownBots rules, and none of them is specific to AI crawlers, so an AI crawler is classified by one of the general rules.
  • On Azure Front Door, bad bots are blocked, good bots are allowed and unknown bots are logged by default, according to Microsoft's WAF overview.
  • Bot Manager 1.1 is available on Front Door Premium; Front Door Standard supports only custom rules.
  • The ruleName field of the Front Door WAF log, or RuleId in the Application Gateway firewall log, shows which Bot rule a crawler matched, and you can override that one rule's action without removing the rule set.
  • The JS challenge action needs a browser that computes the challenge and stores the afd_azwaf_jsclearance or appgw_azwaf_jsclearance cookie, so an AI crawler that fetches raw HTML cannot pass it.
On this page

The Azure WAF Bot Manager rule set sorts bot traffic into three groups, BadBots, GoodBots and UnknownBots, and Microsoft's published rule tables contain no rule for AI crawlers. An AI crawler such as OAI-SearchBot, ClaudeBot or PerplexityBot therefore lands in one of the general rules, and which one is not documented. On Azure Front Door, bad bots are blocked, good bots are allowed and unknown bots are only logged by default, so the outcome depends on the rule it matched and on any action you changed. To fix a blocked crawler, find the matched rule ID in the WAF log and override that one rule, or allow the crawler's published IP ranges with a custom rule.

Allowing a crawler removes a technical barrier. It does not mean an AI tool will use or cite your pages. If you also run AWS, the AWS WAF Bot Control guide covers the equivalent there, and AWS, unlike Azure, has a dedicated AI category.

What the Bot Manager 1.0 and 1.1 rule sets contain

Microsoft lists every rule on its Front Door DRS rule groups page and on the matching Application Gateway rules page. The rule IDs are the same on both products at the time of writing.

Rule IDGroupBot Manager 1.0Bot Manager 1.1
Bot100100BadBotsMalicious bots detected by threat intelligenceSame
Bot100200BadBotsMalicious bots that falsified their identitySame
Bot100300BadBotsNot presentHigh risk bots detected by threat intelligence
Bot200100GoodBotsSearch engine crawlersSearch engine crawlers
Bot200200GoodBotsUnverified search engine crawlersVerified miscellaneous bots
Bot200300GoodBotsNot presentVerified link checker bots
Bot200400GoodBotsNot presentVerified social media bots
Bot200500GoodBotsNot presentVerified content fetchers
Bot200600GoodBotsNot presentVerified feed fetchers
Bot200700GoodBotsNot presentVerified advertising bots
Bot300100UnknownBotsUnspecified identitySame
Bot300200UnknownBotsTools and frameworks for web crawling and attacksSame
Bot300300UnknownBotsGeneral-purpose HTTP clients and SDKsSame
Bot300400UnknownBotsService agentsSame
Bot300500UnknownBotsSite health monitoring servicesSame
Bot300600UnknownBotsUnknown bots detected by threat intelligenceSame, and also IP addresses matched to the Tor network
Bot300700UnknownBotsOther botsSame

Note that Bot200200 changes meaning between versions: unverified search engine crawlers in 1.0, verified miscellaneous bots in 1.1. If you read an old runbook or a log from a policy that still uses 1.0, check the version before acting on the ID. Microsoft says Bot100100 and Bot300600 scan both the client IP and the IPs in the X-Forwarded-For header.

Microsoft's Front Door WAF overview describes the groups. Bad bots include malicious IPs from the Microsoft Threat Intelligence feed and "bots that identify themselves as good bots but their IP addresses don't belong to legitimate bot publishers." Good bots are verified bots in categories such as search engines (it names Googlebot and Bingbot), link checkers, social media and advertising. Unknown bots are user agents without additional validation, plus medium-confidence threat intelligence IPs.

Where AI crawlers fall in the rule set

Searching Microsoft's rule tables, the overview and the Bot Manager 1.1 announcement for AI turns up nothing: there is no AI category and no list of AI crawlers. Microsoft also does not publish which named bots each GoodBots rule recognizes. So the honest answer is that it depends on Microsoft's internal signatures, and your logs are the only reliable source.

The plausible outcomes, and what each means for the crawler:

Matched ruleDefault group action on Front DoorWhat the crawler gets
A GoodBots rule, such as Bot200200 or Bot200500 in 1.1AllowYour page
An UnknownBots rule, such as Bot300700 (Other bots)LogYour page, unless a later rule or a custom rule blocks it
Bot100200, falsified identityBlockA block response
Bot100100 or Bot100300, threat intelligenceBlockA block response

Two cases need care. A real crawler that Azure does not recognize as a verified publisher may sit in UnknownBots, which is harmless at the default Log action but becomes a block if someone set the whole group to Block. And a crawler whose request comes from cloud IPs that also appear in threat intelligence feeds could match a BadBots rule; Microsoft does not document how often that happens. The training, search and user-triggered split helps decide which of these crawlers you want to let in at all.

How rule order and actions work

Three documented behaviors decide what happens to a crawler request on Front Door:

  1. Custom rules run first. Microsoft says custom rules are processed before managed rule sets, and once a rule with Allow or Block matches, lower-priority rules are not processed. A custom Allow rule for a crawler therefore skips Bot Manager. The exception is the HTTP DDoS rule set, which runs before custom rules and is not bypassed by Allow.
  2. Bot Manager does not use anomaly scoring. Microsoft states the anomaly score action is not applicable to the Bot Manager rule set, so a bot rule acts on its own, unlike DRS 2.x rules.
  3. Detection mode only logs. In Detection mode the WAF records the match and takes no other action, which is a safe way to see what a new rule set would do.

The tier matters too. Microsoft says the WAF has full capabilities on Front Door Premium, while Front Door Standard supports only custom rules, and Bot Manager 1.1 is available on Premium. On Application Gateway, bot protection requires the WAF v2 SKU, and Microsoft's bot protection overview says it runs alongside one OWASP rule set as Microsoft_BotManagerRuleSet_1.1.

How to find the matched rule ID in WAF logs

WAF logs are not on by default. Enable them under Diagnostic settings for the Front Door profile or the application gateway, then query Log Analytics.

Azure Front Door

Microsoft's monitoring page lists the fields of FrontDoorWebApplicationFirewallLog. The ones that matter here: RuleName (the rule that matched), Action, PolicyMode, RequestUri, ClientIP and SocketIP, and TrackingReference. Microsoft's sample entry shows a DRS match as Microsoft_DefaultRuleSet-1.1-SQLI-942100, so expect the bot rule ID inside RuleName as well; confirm the exact format against your own rows.

kusto
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.CDN" and Category == "FrontDoorWebApplicationFirewallLog"
| where ruleName_s contains "Bot"
| project TimeGenerated, action_s, ruleName_s, requestUri_s, clientIP_s, socketIP_s, details_matches_s
| order by TimeGenerated desc

The ResourceProvider, Category and action_s names come from Microsoft's sample query. The other column names follow the same _s convention but are not listed on that page, so run getschema on your table if one returns nothing. Microsoft describes Details as the names and values of the request parts that matched, such as header names; it does not say whether a bot match records the User-Agent there, so check a few of your own rows.

Two fields are easy to misread:

  • ClientIP comes from X-Forwarded-For when that header is present. SocketIP is the address Front Door actually saw. Compare SocketIP against the crawler's published IP ranges, as described in verifying real AI crawlers.
  • TrackingReference is the value of the X-Azure-Ref response header. If a crawler operator or a test request gives you that header, you can find the exact log row. Other vendors' markers are covered in which WAF blocked a crawler.

Application Gateway

In resource-specific mode the firewall log goes to the AGWFirewallLogs table, with RuleId, RuleSetType, Action, Message, DetailedData, ClientIp and RequestUri columns:

kusto
AGWFirewallLogs
| where RuleId startswith "Bot"
| project TimeGenerated, Action, RuleId, RuleSetType, Message, DetailedData, ClientIp, RequestUri
| order by TimeGenerated desc

The column reference lists only OWASP for RuleSetType, but Microsoft's own bot protection log example shows MicrosoftBotProtection with the message IPReputationTriggered, so filter on the rule ID rather than the type. Microsoft does not show a bot row with its RuleId, so if the startswith "Bot" filter returns nothing, look at recent rows without it and adjust.

How to change one rule's action without disabling the set

Once you know the rule ID, override that rule only. Turning off the whole rule set would also stop the BadBots protection you want to keep.

On Front Door, the Azure CLI front-door extension has a managed rule override command. This sets Bot300700 to Log, for example after someone set the UnknownBots group to Block:

bash
az network front-door waf-policy managed-rules override add \
  --resource-group my-rg \
  --policy-name MyWafPolicy \
  --type Microsoft_BotManagerRuleSet \
  --rule-group-id UnknownBots \
  --rule-id Bot300700 \
  --action Log

The accepted values for --action are Allow, AnomalyScoring, Block, Log and Redirect at the time of writing. The portal does the same from Managed rules in the WAF policy, where each rule's action can be edited.

On Application Gateway, the Az PowerShell cmdlet New-AzApplicationGatewayFirewallPolicyManagedRuleOverride builds the same override, with -RuleId, -State Enabled and -Action. You then add it to a rule group override for the bot rule set and update the policy.

powershell
$override = New-AzApplicationGatewayFirewallPolicyManagedRuleOverride `
  -RuleId 'Bot300700' -State Enabled -Action Log

When to use a custom Allow rule instead

Overriding a rule changes the outcome for every bot that matches it, including scrapers. If you want to let in one crawler, a custom rule on Front Door that checks both the user-agent token and the source IP is narrower. Microsoft's AS number match page explains why the IP condition should use SocketAddr: RemoteAddr is taken from X-Forwarded-For, which a client can set.

json
{
  "name": "AllowOAISearchBot",
  "priority": 10,
  "ruleType": "MatchRule",
  "matchConditions": [
    {
      "matchVariable": "RequestHeader",
      "selector": "User-Agent",
      "operator": "Contains",
      "transforms": ["Lowercase"],
      "matchValue": ["oai-searchbot"]
    },
    {
      "matchVariable": "SocketAddr",
      "operator": "IPMatch",
      "matchValue": ["<prefixes from OpenAI's searchbot.json>"]
    }
  ],
  "action": "Allow"
}

Load the prefixes from the operator's published list and refresh them on a schedule; AI crawler IP ranges for firewalls lists the files. Never allow on the user agent alone, since anyone can copy the string.

Why AI crawlers cannot pass the JavaScript challenge

Front Door Premium offers a JS challenge action in custom rules and in the Bot Manager 1.x rule set. On Application Gateway the JS challenge is in preview at the time of writing, and Microsoft's Application Gateway rules page lists only Allow, Block and Log as actions for the Bot Manager rule set. Microsoft's JavaScript challenge page explains the mechanism: the client gets a challenge page, the browser computes the challenge and sends the answer to an Azure endpoint, and on success it receives a validity cookie, afd_azwaf_jsclearance on Front Door or appgw_azwaf_jsclearance on Application Gateway. The cookie lasts 30 minutes by default, configurable from 5 to 1,440. Requests that fail the computation are blocked, and the challenge is issued again if the client's IP changes.

A crawler that fetches HTML without running scripts never computes the answer, so it receives the challenge page instead of your content. Microsoft does not document the HTTP status of the challenge page. In logs, Front Door records the outcome as JSChallengeIssued, JSChallengePass, JSChallengeValid or JSChallengeBlock; a crawler that only ever produces JSChallengeIssued rows is not getting through. For how challenge pages look to crawlers on other vendors, see WAF challenge pages and AI crawlers.

If a bot rule that matches crawlers you want is set to JS challenge, change it back to Allow or Log, or add a custom Allow rule like the one above, which runs before Bot Manager.

Frequently asked questions

Does Azure WAF block GPTBot or ClaudeBot by default?

Microsoft's documentation does not name any AI crawler or give it its own rule. With default actions on Front Door only BadBots rules block, so a block means the crawler matched a BadBots rule or someone changed an action; the WAF log shows which.

Which Bot Manager version should I use?

Microsoft's Front Door setup guide says it is usually good practice to use the most recent version, which is 1.1 at the time of writing. It splits good bots into more verified categories, so you can tell a content fetcher from a search engine crawler in your logs.

Can I allow a crawler without turning off Bot Manager?

Yes. Add a custom Allow rule that matches the crawler's user-agent token and its published IP ranges with SocketAddr. Custom rules run before the managed rule set, so the request skips Bot Manager while every other bot is still checked.

Why does my curl test with a crawler user agent give a different result?

Your request comes from your own IP, not the operator's, so Azure cannot verify it as that bot and may treat it as unknown or as a falsified identity. Use the WAF log for the real crawler's outcome.

Check what a crawler sees

After changing a rule, query the WAF log again for the crawler's user agent and confirm the action is Allow or Log. To see what one public page returns to a client with no special treatment, the free check needs no account. It runs from Crawlito's servers, so a WAF that judges by IP may treat it differently from the real crawlers.

Filed under CDN and bot protection. Topics: bot-protection, waf, azure, ai-crawlers, troubleshooting