AWS WAF HostingProviderIPList: does it block AI crawlers?
AWS WAF HostingProviderIPList blocks requests from hosting and cloud providers, where AI crawlers run. How to confirm it and exempt verified crawler IP ranges.
By Crawlito · 8 min read
In short
- HostingProviderIPList is a rule in the AWSManagedRulesAnonymousIpList group with a default Block action, aimed at web hosting and cloud provider addresses, which are where AI crawlers run.
- AWS says the hosting provider list does not include AWS IP addresses, so crawlers on Microsoft Azure or Google Cloud are the likelier casualties than crawlers that run on AWS.
- You cannot remove an address from the list; the documented fixes are a scope-down statement on the rule group or a Count override plus a label-match rule that runs after it.
- Override the single rule to Count with RuleActionOverrides, not the whole rule group, so AnonymousIPList keeps blocking and the HostingProviderIPList label reaches your own rule.
- Build the exemption from the operators' published IP range files, refreshed on a schedule, never from the User-Agent header.
On this page
Yes, it can. HostingProviderIPList is a rule in the AWS WAF Anonymous IP list managed rule group, and it blocks requests from web hosting and cloud provider addresses by default. AI crawlers such as GPTBot, OAI-SearchBot and ClaudeBot fetch from cloud data centers, so a web ACL with this rule group can return 403 to them while visitors on home and mobile connections get through. The fix AWS documents is to keep the rule group, set that one rule to Count, and add a rule after it that blocks on its label unless the request comes from an IP set you trust.
This post covers that one rule. If you also run Bot Control, its separate CategoryAI behavior is explained in AWS WAF Bot Control and AI crawlers. For country and ASN rules that catch crawlers, see geo blocking and IP allowlists.
What HostingProviderIPList does
The rule group is AWSManagedRulesAnonymousIpList, vendor AWS, with a capacity of 50 WCUs. AWS's IP reputation rule group documentation says it contains rules to block requests from services that hide the viewer's identity, including VPNs, proxies, Tor nodes and web hosting providers. It has two rules:
| Rule | Default action | What it matches | Label |
|---|---|---|---|
AnonymousIPList | Block | Tor nodes, temporary proxies and other masking services | awswaf:managed:aws:anonymous-ip-list:AnonymousIPList |
HostingProviderIPList | Block | Web hosting and cloud providers, "less likely to source end-user traffic" | awswaf:managed:aws:anonymous-ip-list:HostingProviderIPList |
Three details from the same page decide how this affects crawlers:
- AWS addresses are excluded. The documentation states that the hosting provider list does not include AWS IP addresses.
- The rule reads the connecting address. These rules use the source IP of the web request origin. Behind another proxy or load balancer, that is the last proxy's address, not the client's.
- No versions or notifications. AWS says the IP reputation groups provide no versioning or SNS update notifications, so the list can change without you hearing about it.
AWS does not publish which ranges are on the list, and it does not say whether search engine crawlers are treated differently. Treat any crawler that fetches from a hosting or cloud network as a possible match until your own logs say otherwise.
Which AI crawlers are likely to match
The operators publish their crawler ranges, and you can see who owns them. When we looked up sample addresses from each file in the ARIN and RIPE registry records (RDAP) on 15 November 2026, the owners were:
| Crawler list | Sample address owner | Likely affected by HostingProviderIPList |
|---|---|---|
| OpenAI gptbot.json, searchbot.json | Microsoft | Yes, possibly: not an AWS address |
| Anthropic bots.json | Google Cloud for most /32 entries, plus one /22 whose registry record is named AWS-ANTHROPIC | Possibly, for the Google Cloud addresses |
| Perplexity perplexitybot.json | Amazon | Less likely: AWS says its own addresses are excluded |
That is an inference from ownership, not a list of confirmed matches, and the files change often. The only way to know is to look at what the rule actually did to crawler requests on your web ACL. The geo-blocking post has a fuller breakdown by network and country if you also filter by location.
How to confirm the rule is blocking crawlers
Check sampled requests
The quickest look is the web ACL's sampled requests. GetSampledRequests returns up to 500 requests picked at random from the first 5,000 in a time window, and the window can be anywhere in the previous three hours, per the CLI reference. The --rule-metric-name is the MetricName in the rule group's VisibilityConfig in your web ACL, such as the AWS-AWSManagedRulesAnonymousIpList used in the JSON further down.
aws wafv2 get-sampled-requests \
--web-acl-arn "$WEB_ACL_ARN" \
--rule-metric-name AWS-AWSManagedRulesAnonymousIpList \
--scope REGIONAL --region eu-west-1 \
--time-window StartTime=2026-11-15T09:00Z,EndTime=2026-11-15T12:00Z \
--max-items 500 \
--query 'SampledRequests[?RuleNameWithinRuleGroup==`AWS#AWSManagedRulesAnonymousIpList#HostingProviderIPList`].[Action,Request.ClientIP,Request.Headers[?Name==`User-Agent`].Value|[0]]' \
--output text > sampled.txtFor a CloudFront web ACL, use --scope CLOUDFRONT --region us-east-1. Each line of sampled.txt shows the action, the client IP and the user agent, separated by tabs. A BLOCK with GPTBot or OAI-SearchBot in the user agent is the pattern to look for. The header filter is case-sensitive, so if the user agent column comes back as None, check whether your requests record the header as user-agent.
Match the IPs against the published lists
A user agent proves nothing on its own, since anyone can send it. Check whether each blocked IP sits inside a published range. These commands download the lists with curl, then read IPs from standard input and print the list each one belongs to:
mkdir -p lists
curl -fsS -o lists/openai-searchbot.json https://openai.com/searchbot.json
curl -fsS -o lists/openai-gptbot.json https://openai.com/gptbot.json
curl -fsS -o lists/openai-chatgpt-user.json https://openai.com/chatgpt-user.json
curl -fsS -o lists/anthropic.json https://claude.com/crawling/bots.json
curl -fsS -o lists/perplexitybot.json https://www.perplexity.ai/perplexitybot.json
cat > match_crawler_ips.py <<'EOF'
import glob, ipaddress, json, os, sys
nets = []
for path in glob.glob("lists/*.json"):
name = os.path.basename(path)[:-5]
with open(path) as f:
for p in json.load(f)["prefixes"]:
nets.append((name, ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))))
for line in sys.stdin:
ip = ipaddress.ip_address(line.split()[0])
hits = [n for n, net in nets if ip.version == net.version and ip in net]
print(ip, ",".join(hits) or "not in any list")
EOF
cut -f2 sampled.txt | python3 match_crawler_ips.pyAn IP that matches a list and was blocked by HostingProviderIPList is a real crawler being turned away. A crawler user agent from an IP in no list is an impostor, and the block is working as intended. Verifying real AI crawlers covers the reverse DNS method for operators that use it.
Read the full logs for a longer window
Sampled requests only reach back three hours. With web ACL logging on, the log fields show the outcome for every request: action, terminatingRuleId, the client IP in httpRequest, labels, and inside ruleGroupList the terminatingRule and nonTerminatingMatchingRules. After you switch the rule to Count, matches appear as non-terminating with the original action in overriddenAction, which lets you keep measuring how many crawler requests it would have blocked.
How to exempt verified crawlers
AWS's knowledge center article is clear on the starting point: you cannot remove an IP address from the Anonymous IP list. It documents two ways around it, both built on an IP set of the addresses you trust.
Step 1: build IP sets from the published ranges
An IP set holds up to 10,000 addresses or ranges, according to the IP set match documentation, which is far more than the crawler lists need. Each IP set has a single address version, IPV4 or IPV6, set when you create it. At the time of writing, the OpenAI, Anthropic and Perplexity files listed above contain only IPv4 prefixes, so one IPv4 set is enough; add an IPv6 set if that changes. Include only the crawlers you want to let through, and regenerate the sets on a schedule, since the files change. The approach for keeping such lists current is in AI crawler IP ranges for firewall allowlists.
aws wafv2 update-ip-set \
--name ai-crawlers-v4 --scope REGIONAL --region eu-west-1 \
--id "$IP_SET_ID" --lock-token "$LOCK_TOKEN" \
--addresses $(cat crawler-v4.txt)UpdateIPSet replaces the whole address list with the one you send, and needs the current lock token from get-ip-set.
Step 2, option A: override the rule to Count and act on its label
This is the label method from the knowledge center article. First override only HostingProviderIPList inside the group. The override documentation says a rule action override makes matching requests behave as if the rule's action were Count, which is non-terminating, so evaluation continues and the label stays on the request:
{
"Name": "AWS-AWSManagedRulesAnonymousIpList",
"Priority": 20,
"OverrideAction": { "None": {} },
"Statement": {
"ManagedRuleGroupStatement": {
"VendorName": "AWS",
"Name": "AWSManagedRulesAnonymousIpList",
"RuleActionOverrides": [
{ "Name": "HostingProviderIPList", "ActionToUse": { "Count": {} } }
]
}
},
"VisibilityConfig": {
"SampledRequestsEnabled": true,
"CloudWatchMetricsEnabled": true,
"MetricName": "AWS-AWSManagedRulesAnonymousIpList"
}
}Then add your own rule with a higher priority number, so it runs after the group, that blocks requests carrying the label unless they come from your IP set:
{
"Name": "block-hosting-providers-except-crawlers",
"Priority": 30,
"Action": { "Block": {} },
"Statement": {
"AndStatement": {
"Statements": [
{ "LabelMatchStatement": { "Scope": "LABEL", "Key": "awswaf:managed:aws:anonymous-ip-list:HostingProviderIPList" } },
{ "NotStatement": { "Statement": { "IPSetReferenceStatement": { "ARN": "arn:aws:wafv2:eu-west-1:111122223333:regional/ipset/ai-crawlers-v4/EXAMPLE" } } } }
]
}
},
"VisibilityConfig": {
"SampledRequestsEnabled": true,
"CloudWatchMetricsEnabled": true,
"MetricName": "BlockHostingProvidersExceptCrawlers"
}
}If you add an IPv6 set, put both IPSetReferenceStatement entries in an OrStatement inside the single NotStatement. Two separate NotStatement entries joined by OrStatement would be true for almost every request, because an IPv4 crawler is never in the IPv6 set. AWS's labeling guide describes exactly this pattern for managed rule groups: count the rule, then handle the request after the group based on its label. A label only exists for rules that run later, so the label rule must have the larger priority number.
Step 2, option B: a scope-down statement
The knowledge center's other method adds a scope-down statement to the managed rule group so it only inspects requests that do not originate from your IP set. It is one change instead of two, but it exempts those IPs from AnonymousIPList as well. AWS's scope-down documentation says requests that do not match the scope-down statement are treated as not matching the rule group, so you get no labels from the group for them. For ranges taken from an operator's own crawler file, that trade-off is often acceptable.
Mistakes to avoid
- Counting the whole group. Overriding the rule group's return action to Count is a different setting. AWS notes it does not change how the group evaluates requests and is not a good option for testing individual rules, and it also stops
AnonymousIPListfrom blocking anything. UseRuleActionOverrideson the single rule. - An Allow rule placed after the group. With the rule still on Block, the request ends at the group and never reaches your Allow rule.
- A proxy in front of AWS WAF. If the web ACL sees a proxy address, the rule matches the proxy, not the crawler. The IP set statement can read a header such as
X-Forwarded-Forthrough its forwarded IP configuration, with a choice of first, last or any address and a fallback for malformed values, but only trust a header your own proxy sets. - Exempting by user agent. A rule that skips the block for
GPTBotin theUser-Agentlets every scraper through with one header change.
Should you keep HostingProviderIPList at all?
It is a reasonable rule for login, checkout and API paths, where requests from data centers are rarely real customers. For public content you want AI search tools to read, it blocks the very networks the crawlers use. One compromise is a second label rule that blocks every labelled request on those sensitive paths, with no IP set exemption, while the rule above handles the rest of the site. Keep in mind that other layers can still block the same requests, so if crawler errors persist after the change, work out which WAF or CDN layer blocked the crawler from the response headers.
Removing this block lets crawlers fetch your pages. It does not decide whether an AI tool uses or cites them.
Frequently asked questions
Does HostingProviderIPList block Googlebot?
AWS does not document whether Google's crawler addresses are on the list. Check your sampled requests or logs for Googlebot requests terminated by this rule, and exempt Google's published ranges the same way if you find any.
Can I ask AWS to remove an IP from the Anonymous IP list?
AWS's knowledge center article says you cannot directly remove IP addresses from the list. Use a scope-down statement or a Count override with a label-match rule instead.
Is it safe to set HostingProviderIPList to Count?
Count on its own lets all hosting-provider traffic through. Pair it with the label-match rule so only your allowlisted ranges are exempt and everything else that matched is still blocked.
Why do crawlers that run on AWS still get blocked?
Not by this rule, if AWS's statement that its own addresses are excluded holds for your traffic. Look at terminatingRuleId in the log: another rule, such as Bot Control or a rate rule, is probably responsible.
Check what a crawler gets from your site
After you change the rule, run the free check on an important page to see the status code and any block page it returns. The check sends its own requests, not GPTBot's, so a rule keyed to specific crawler IPs can treat it differently, which is why the WAF logs remain the evidence. For a full site, the audit covers up to 500 pages and comes as a PDF with evidence and prioritized fixes.
Filed under CDN and bot protection. Topics: aws, waf, bot-protection, ai-crawlers, troubleshooting