The requested URL was rejected: F5 BIG-IP blocking crawlers
The requested URL was rejected with a support ID is the F5 BIG-IP WAF block page. Find the log entry, the Bot Defense class and the DNS fault behind it.
By Crawlito · 9 min read
In short
- A page that says the requested URL was rejected and gives a support ID is the blocking response of F5 BIG-IP Application Security Manager (ASM) or Advanced WAF, which issues response pages only when its policy is in Blocking mode.
- Every default F5 response page carries a support ID, and the same ID appears in the request details under Security > Event Logs > Application > Requests, next to the violations that triggered the block.
- In the Bot Defense profile templates, Trusted Bot is set to Alarm in Relaxed, Balanced and Strict, while Unknown clients are rate limited in Balanced and blocked in Strict.
- Bot Defense relies on reverse DNS lookups, and F5 bug 957321 documents that an invalid first DNS resolver stops it verifying trusted bots, so valid search engines are classified as malicious bots.
- F5 does not document whether AI crawlers such as GPTBot or ClaudeBot are recognised as trusted bots, so read the Bot Defense request log to see how they are classified on your device.
On this page
- What the requested URL was rejected message means
- How to confirm the F5 block page with curl
- How to look up the support ID in the BIG-IP event log
- How Bot Defense classifies crawlers
- When trusted bots fail DNS verification
- How to fix F5 blocks for crawlers you want
- Frequently asked questions
- Check what crawlers receive from your site
"The requested URL was rejected. Please consult with your administrator." followed by "Your support ID is" and a long number is the block page of F5 BIG-IP Application Security Manager (ASM) and Advanced WAF. It means a security policy on the BIG-IP in front of the site refused the request before it reached the web server. The support ID is the key: an administrator can look it up in the BIG-IP event log and see exactly which violation or bot rule fired. When a crawler gets this page, it receives that one sentence instead of your content.
This guide covers what is specific to F5: how to recognise the page, how the support ID leads to the log entry, how the Bot Defense profile classifies crawlers, and two documented faults that block real search engines. For the general 403 workflow, see blocked due to access forbidden (403). For JavaScript challenges, see WAF challenge pages and AI crawlers.
What the requested URL was rejected message means
F5's manual chapter on what happens if a request is blocked explains the mechanism. The security policy has a default blocking response page that it returns when a request violates the policy. Three facts from that chapter matter when you troubleshoot:
- Response pages are only sent in Blocking mode. A policy in Transparent mode logs violations but lets the request through, so if you see this page, enforcement is on.
- Every default response page contains a support ID. The page template holds the variable
<%TS.request.ID()%>, which the system replaces with a support ID number when it issues the page. F5 says customers can use the ID to identify the request. - The page can be customised. Under Security > Application Security > Policy > Response Pages, the Default Response Page tab offers Default Response, Custom Response, Redirect URL, SOAP Fault and Erase Cookies. There are separate AJAX and XML response pages, and a login response page for login violations.
F5's response page chapter does not reproduce the text of the default page, so treat the sentence above as the wording commonly seen on it rather than a documented string. The wording also varies. Many sites keep the default sentence, others replace it with their own branding but keep the support ID. If a block page shows a long numeric support ID and nothing else useful, F5 is the likely source. Which WAF blocked the crawler compares the markers of other vendors, and Akamai's Access Denied page is the equivalent with a reference number.
Bot Defense, the bot profile that sits beside the security policy, has its own response page with its own support ID command, %BOTDEFENSE.support_id%, as F5 bug 1690593 shows. A crawler can therefore be stopped either by a policy violation or by bot classification, and the two are logged in different places.
How to confirm the F5 block page with curl
Fetch the page as a browser and as a crawler, and look at the body rather than only the status code:
url="https://example.com/page"
for ua in "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"; do
curl -s -A "$ua" -o body.html -w "%{http_code} %{size_download} bytes\n" "$url"
grep -io "support id[^<]*" body.html | head -1
doneA short body with a support ID line is the F5 page. Do not judge by the status code alone. F5's response page documentation does not state which status the default page uses, and administrators can change it, so a block can arrive as 200 OK as well as 4xx. A crawler that sees 200 with one sentence of text has still received nothing of yours.
The limit is the same as for any curl test: your request does not come from the crawler's IP addresses. A rule that keys on the user agent shows up here. A rule that depends on IP reputation, reverse DNS or request rate does not, and a block that hits the real crawler may not reproduce from your laptop. The event log is the evidence.
How to look up the support ID in the BIG-IP event log
Send the support ID to whoever runs the BIG-IP. F5's reporting and monitoring chapter gives the path:
- On the Main tab, click Security > Event Logs > Application > Requests. By default this shows illegal requests for all security policies.
- Filter by the support ID. F5 bug 666986 notes that on some older versions the filter failed for IDs shorter than 19 digits, and suggests filtering by the last four digits as a workaround.
- Click the request. The Request Details area lists the violations, the security policy, the support ID, the violation rating and potential attacks.
- Click HTTP Request to see what the client sent, including the user agent and source IP.
If the ID is not in the Application request log, check Security > Event Logs > Bot Defense > Requests, which F5's Bot Defense chapter describes as the per-request bot log, including an "Enforced By" field. Bot Defense logging needs a logging profile with Bot Defense enabled, attached to the virtual server, and F5 recommends sending it to a remote logger rather than logging locally. Security > Event Logs > Bot Defense > Bot Traffic summarises bot traffic across the configured virtual servers.
Before you change anything, check that the blocked client is real. Copy the source IP from the request details and verify it with the operator's method, as described in how to verify real AI crawlers. A request that only borrows a crawler's user agent should stay blocked.
How Bot Defense classifies crawlers
The Bot Defense profile sorts every client into a class and applies an action per class. The profile templates set these defaults, from F5's Bot Defense chapter for BIG-IP 17.5:
| Class | Relaxed | Balanced | Strict |
|---|---|---|---|
| Trusted Bot | Alarm | Alarm | Alarm |
| Untrusted Bot | Alarm | Alarm | Block |
| Suspicious Browser | Alarm | CAPTCHA | Block |
| Malicious Bot | Block | Block | Block |
| Unknown | None | Rate limit | Block |
The chapter does not define Alarm, but it is the non-blocking action: F5's own workaround for search engines being challenged, in bug 746750 below, is to set Trusted Bot to Alarm. A technical article on F5's DevCentral community about the bot profile defines the classes: Trusted Bots are detected with search engine signatures, Untrusted Bots with signatures for untrusted bots, Malicious Bots with malicious bot signatures or malicious behavioural anomalies, and Unknown covers clients detected by neither signatures nor anomalies.
What this means for crawlers:
- A client classed as a Trusted Bot is not blocked under any template, because Trusted Bot is set to Alarm in all three.
- A client matched by an untrusted bot signature is blocked under Strict. F5 does not publish which AI crawlers its signatures cover or which class they fall in, so do not assume GPTBot, ClaudeBot, OAI-SearchBot or PerplexityBot are trusted. Read their class in the Bot Defense request log.
- A client with no matching signature is Unknown, which Balanced rate limits and Strict blocks.
Browser verification adds another barrier. The chapter says the system sends a JavaScript challenge on first access, requests that do not answer it are not sent to the server, and requests to non-HTML URLs without the cookie are dropped and treated as bots. The Strict template also requires browsers to pass verification before access. A crawler that does not run JavaScript cannot answer the challenge, so if it ends up in a class that is verified or challenged, it gets nothing.
Two settings can widen blocking further. DoS Attack Mitigation Mode, on by default in the Balanced and Strict templates and active only when DoS protection is attached to the same virtual server, blocks every class except Trusted Bot (which stays at Alarm) and verifies browsers before access during an attack. New signatures from Live Update go to staging first, where matching requests are logged but not mitigated until someone enforces them under the Signature Enforcement tab, so a crawler that was fine last month can be blocked after enforcement.
When trusted bots fail DNS verification
User agents are easy to fake, so a client claiming to be a search engine has to be verified. The Bot Defense chapter says the mechanism uses reverse lookup, so it needs a DNS server under System > Configuration > Device > DNS and a DNS resolver under Network > DNS Resolver > DNS Resolver List, and the resolver must use the default route domain.
When that lookup cannot work, real crawlers lose their trusted status. F5 documents this in bug 957321:
- Symptom: when the first DNS resolver in the list is invalid, Bot Defense cannot verify trusted bots and classifies them as malicious bots.
- Impact: valid search engines are wrongly classified as malicious, and blocked if enforcement is on. Malicious Bot is blocked in all three templates.
- Workaround: fix the first DNS resolver in the list. F5 notes it may be the built-in resolver
f5-aws-dns. - Versions: affected versions listed run from 15.1.5 to 15.1.10.8, and no fix version is listed at the time of writing.
An older fault, bug 746750, affected BIG-IP 14.1.0 and 14.1.0.1: with one of the predefined profiles attached, search engines could get Device ID challenges and, since they cannot run JavaScript, would most likely be blocked. It was fixed in 14.1.0.2 and 15.0.0. The documented workaround was to set Trusted Bot to Alarm on the profile's Mitigation Settings tab.
The pattern to look for in the log is a request whose source IP passes the operator's own verification but which Bot Defense lists as a Malicious Bot. That points at verification, not at the crawler.
How to fix F5 blocks for crawlers you want
Work from the log entry, and keep each change narrow:
- Fix DNS first. If verified search engines show up as malicious, check the device DNS and resolver configuration and confirm that the first resolver in the list works. No allow rule fixes a verification that cannot run.
- Check the Trusted Bot action. Confirm that it is still Alarm on the profile's Bot Mitigation Settings tab, the default in every template.
- Handle AI crawlers explicitly. If a crawler you want, such as OAI-SearchBot, is Unknown or Untrusted under a strict profile, add a Bot Defense whitelist entry. The Whitelist tab holds sources, URLs or source and URL pairs that the system does not check for bot attacks; F5 says whitelisted sources are trusted and never blocked, entries apply in list order, and each whitelist belongs to one profile. Scope it to the operator's published IP ranges, not to a user agent, which anyone can copy.
- Fix policy violations at the violation. If the support ID points to an Application Security violation rather than Bot Defense, adjust the setting behind that violation, for example the URL or the request rule involved. Do not switch the whole policy to Transparent.
- Retest and watch the logs. Rerun the curl test, then watch the Bot Defense request log for the next real visit from each crawler.
Decide which crawlers you want first. Training, search and user crawlers explains the difference between them.
Frequently asked questions
Is the requested URL was rejected an error in my website code?
No. The page is generated by the F5 BIG-IP in front of the web server, and the request never reached your application. The fix is in the BIG-IP security policy or Bot Defense profile.
What can I do with the support ID if I am not the site administrator?
Send it to the site owner with the time and the URL you requested. Only someone with access to the BIG-IP can look it up in the event log and see why the request was blocked.
Does F5 Bot Defense allow Googlebot and Bingbot by default?
Search engines detected by signature are Trusted Bots, which every template sets to Alarm, so they are not blocked once their identity checks out. If DNS verification fails, F5 bug 957321 says they can be classified as malicious and blocked instead.
Are GPTBot and ClaudeBot trusted bots in F5?
F5 does not document how its bot signatures classify AI crawlers. Check the Bot Defense request log on your device, which shows the class and action for each request.
Will robots.txt stop the F5 block?
No. robots.txt tells a crawler what it may fetch, but the BIG-IP answers before the request reaches your server. If it blocks the crawler, robots.txt allowing it changes nothing.
Check what crawlers receive from your site
After fixing the policy, run the free check on an important page to see whether the main AI crawlers can fetch it and whether the response holds your content or a block page. It sends requests from Crawlito's servers, not from the crawlers' networks, so the BIG-IP event log remains the evidence for what a real crawler received.
Filed under CDN and bot protection. Topics: bot-protection, waf, troubleshooting, http-status