Skip to content

Imunify360 bot protection blocking crawlers: what to ask a host

How Imunify360 bot protection greylists, challenges and rate-limits crawlers, how to recognise each response, and the exact changes to request from your host.

By Crawlito · 10 min read

In short

  • Imunify360 puts suspicious IPs on a Gray List, and web requests for HTML from a greylisted IP get a JavaScript Anti-bot Challenge that needs cookies and JavaScript to pass.
  • A greylisted request whose Accept header does not start with text/, such as curl's default /, gets HTTP 415 Unsupported Media Type instead of the challenge page.
  • Imunify360 keeps a server-wide list of whitelisted search engine crawlers by domain, and its documentation says Anti-bot protection does not block legitimate bots such as Google's crawler.
  • AI Bot Management in the Imunify Security WordPress plugin allows verified AI crawlers 10 requests per minute on the default Balanced preset and 3 on Strict, and answers excess requests with HTTP 429 and Retry-After.
  • On shared hosting the whitelist, CDN detection and most WebShield settings belong to the server administrator, so the fix is usually a precise request to your host.
On this page

Imunify360 can stop a crawler in three ways: it greylists the crawler's IP and sends it to a JavaScript Anti-bot Challenge, it blocks the request outright with a 403, or, on WordPress sites with AI Bot Management switched on, it rate-limits the crawler with a 429. The challenge catches AI crawlers, which are not documented to run scripts. On shared hosting you usually cannot change any of this yourself, so the practical fix is to identify which response the crawler got and ask your host for one specific change: a whitelist entry, an allowlist rule, or correct IP detection behind your CDN.

The general mechanics of challenge pages are covered in why a WAF challenge page shows AI bots nothing, and the WordPress plugin side in WordPress security plugins that block crawlers. This post sticks to what is specific to Imunify360, based on CloudLinux's Imunify360 documentation read on 15 November 2026.

How Imunify360's Gray List and Anti-bot Challenge work

Imunify360 is a server security suite from CloudLinux that runs on cPanel, Plesk and DirectAdmin servers, which is why it turns up on shared hosting. Its web-facing part is WebShield, which inspects HTTP requests before they reach the site, either as a module loaded into the web server or as a reverse proxy, depending on the setup. Three IP lists matter for crawlers:

ListWhat the documentation says it does
White ListIPs that "will not be blocked in any case"; a whitelisted domain shows no Anti-bot Challenge even to a greylisted IP
Gray ListIPs sent to the Anti-bot Challenge on web ports and blocked completely on non-web ports such as SSH and FTP
Black ListIPs blocked; a drop entry responds with 403 on web ports, even through a proxy

There is one documented exception to "not blocked in any case": an IP that is listed in Imunify's own RBL is still dropped at the WAF level with a 403 even when it is on the White List, until the administrator runs imunify360-agent create-rbl-whitelist to sync the firewall whitelist with the WAF.

An IP lands on the Gray List when Imunify360's rules flag its behaviour, for example when Enhanced DOS Protection sees a request flood or after repeated ModSecurity incidents from the same IP. From then on, every connection from that IP to a website on the server is redirected to the challenge page, which Imunify360 also calls the Splash Screen. A browser that runs the script is removed from the Gray List and carries on. Imunify360 adds that repeated violations put the IP back on the list.

The documentation is explicit about the requirement: "Cookies and JavaScript support are required in a browser to successfully pass the challenge." OpenAI, Anthropic and Perplexity do not document their crawlers running JavaScript, so assume a greylisted crawler IP receives the challenge page for every HTML URL until someone removes it from the list or the entry expires.

Two further WebShield features work on traffic rather than on the IP lists. Both are off by default and switched on by the server administrator, who can also let site owners manage their own rules:

  • Under Attack Mode serves "a lightweight JavaScript splash challenge" to every request that matches a domain (and optional path) rule. The documentation says it never blocks outright, that whitelisted IPs are never challenged, and that legitimate automation that cannot solve a JavaScript challenge, search engine crawlers included, should be whitelisted. At the time of writing it is documented for cPanel with Apache or Nginx only.
  • Rate Limiting (the L7 Rate Limiter) counts requests per client IP over a sliding window. For clients over the limit the action can be continue (log only), allow, deny (403) or splash (the JavaScript challenge). The documented initial default is 300 requests per minute with splash.

How Imunify360 says it recognises search engine bots

The documentation states that the Anti-bot protection feature "will not block legitimate bots (e.g., Google crawler)". The mechanism behind that is a server-wide list managed from the command line:

bash
imunify360-agent whitelisted-crawlers list
imunify360-agent whitelisted-crawlers add yandex.com google.com

The list output has description, domains and ID columns. The example in the CLI reference shows Google with the domains .google.com and .googlebot.com. The reference does not explain how a request is matched to those domains.

The documentation does not list which AI crawlers, if any, are in the default whitelisted-crawlers list, and this is a server command, not something a hosting customer can run. Do not assume OAI-SearchBot or ClaudeBot are covered: their operators publish IP ranges for verification, not hostnames. Ask your host to run whitelisted-crawlers list and tell you what is in it.

AI Bot Management: the WordPress rate limits

Imunify360 can also run AI Bot Management (shown as Bot Protection in WordPress) through the Imunify Security plugin for WordPress. At the time of writing it needs Imunify360 8.13.6-6 and plugin wp-4.0.2-2. It is off by default: a host enables it server-wide with the ai_bot_protection key and picks a default preset, and a WordPress administrator can then switch it off or change the preset for their own site. It classifies each request, confirms clients that claim to be a known crawler with forward-confirmed reverse DNS, and applies a per-minute allowance to each class. The site owner guide gives these limits:

ClassBalanced (default)StrictMonitor only
Verified search engines300300Not enforced
Verified AI crawlers103Not enforced
Verified SEO crawlers6020Not enforced
Unknown automated clients52Not enforced
Unverified bots21Not enforced
Malicious botsBlockedBlockedNot enforced

Requests over the limit get HTTP 429 with a Retry-After hint, malicious bots get 403, and clients that keep exceeding their limit can be escalated to a temporary block. IPs on the server's White List are never limited. The guide does not name the AI crawlers it verifies, does not describe a verification method specific to AI crawlers, and does not say whether the count is per IP or per bot. On ImunifyAV and ImunifyAV+ the feature always runs as Monitor only. A crawler that claims a known user agent but fails verification falls into "unverified bots", with 2 requests per minute on Balanced.

Ten requests a minute is tight for a crawler working through a large site. To exempt a specific crawler, use the allowlist described below.

How to recognise an Imunify360 block in a response

Imunify360 does not document a status code or marker header for the challenge page itself, so read the status, the headers and the body together:

What you seeLikely cause
A short HTML page that needs JavaScript and cookies (status not documented)The IP is greylisted and got the Anti-bot Challenge, or Under Attack Mode is on
415 Unsupported Media TypeA greylisted IP sent an Accept header that does not start with text/
403A Black List drop entry, a WAF rule, an IP on Imunify's RBL, Rate Limiting with deny, or AI Bot Management treating the client as malicious
429 with Retry-AfterAI Bot Management rate limit on a WordPress site

The 415 row catches many testers out. The documentation says that for greylisted IPs, a request whose Accept header does not start with text/ gets "HTTP 415 Unsupported Media Type" instead of the challenge, and gives application/json and */* as examples. curl sends Accept: */* by default, so a plain curl test from a greylisted IP shows a 415 while a browser on the same IP sees the challenge.

To test, request a page twice with a crawler user agent, once with curl's default Accept and once with text/html:

bash
UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot'
curl -s -o /dev/null -D - -A "$UA" https://example.com/ | grep -iE '^(HTTP|retry-after|server)'
curl -s -A "$UA" -H 'Accept: text/html' https://example.com/ | head -c 600

A 415 on the first request and a short JavaScript page on the second means your IP is greylisted. A full page on both means it is not, but the test proves only what happens to your IP: the real crawler connects from OpenAI's network, which Imunify360 may treat differently. Run it from a cloud server as well, and see your page as an AI crawler with curl for the wider set of checks. If several security layers sit in front of the site, which WAF blocked the crawler helps tell them apart.

When a CDN makes Imunify360 greylist the wrong IP

Behind Cloudflare or another CDN, every request reaches the server from a CDN address. If Imunify360 reads that address instead of the visitor's, one bad visitor can get a CDN IP greylisted, and then crawlers and people arriving through that CDN node get the challenge too. The documentation says Imunify360 greylists and blocks real IPs behind supported CDNs by reading the CF-Connecting-IP and X-Forwarded-For headers, with CDN support switched on through the known_proxies_support setting (shown as "Detect IPs behind CDN" in the dashboard). The supported list includes Cloudflare, Fastly, CloudFront, BunnyCDN, QUIC.cloud and Sucuri WAF, among others.

The web server also has to trust WebShield as an internal proxy, through mod_remoteip on Apache or ngx_http_realip_module on Nginx. Without it, the documentation says, logs show the local server IP for visitors coming through a CDN. A proxy subnet that is not on the supported list can be added in CIDR notation:

bash
imunify360-agent remote-proxy add 192.0.2.0/24 --name "my_proxy"

The documentation also warns that the Anti-bot Challenge "is not compatible with aggressive CDN caching modes", naming Cloudflare Browser Cache TTL and Cache Everything with Edge Cache TTL. If the CDN caches the challenge page, later visitors get the cached challenge whether they passed it or not, and a crawler gets it too. It recommends not combining Cache Everything with Edge Cache TTL and setting Browser Cache TTL to "Respect Existing Header".

What to ask your host

The White List, the whitelisted crawlers list, CDN detection and the server-wide Under Attack Mode and Rate Limiting switches are administrator controls. The Rate Limiting and Under Attack Mode tabs only appear in your own panel if your hosting provider has made them available to you. A support ticket gets resolved faster when it names the exact change. Include the URL, the time of the failed request, the crawler user agent, its source IP from your access log, and the status code you saw, then ask for one of these:

  1. Check the Gray List for the crawler's IP and remove it. Imunify360 lets the administrator remove any IP from the Gray List and add it to the White List.
  2. Whitelist the crawler's published ranges, not single IPs. OpenAI publishes its crawler IPs in its bot documentation, and Anthropic and Perplexity publish equivalent files; verifying real AI crawlers lists them. Ask for a comment on each entry so the next technician knows why it is there. If the IP still gets a 403 after whitelisting, ask whether it is on Imunify's RBL and whether create-rbl-whitelist has been run.
  3. Show the whitelisted crawlers list, and ask whether the search crawlers you care about can be added.
  4. Confirm "Detect IPs behind CDN" is on and that the web server trusts WebShield's forwarded IP, if your site is behind a CDN.
  5. Check the AI Bot Management preset, if your site runs WordPress: you can change it yourself from the Bot Protection row of the Imunify Security dashboard widget, and the Bot Traffic page (Imunify Security, then Bot Traffic) shows how many bot requests were rate-limited (429) or blocked (403) in the last 24 hours. Its allowlist accepts IPs, CIDR ranges, REST API routes, paths and user agent text; prefer IP ranges, because a user agent match lets through anyone who copies the string.

Avoid asking for Imunify360 or WebShield to be switched off for your account. Allowlisting by verified IP fixes the crawler problem without removing the protection. If the host's own resource limits are the cause instead, the symptoms and fixes are different: see 508 Resource Limit Reached.

Frequently asked questions

Does Imunify360 block Googlebot?

Its documentation says Anti-bot protection does not block legitimate bots such as Google's crawler, and the whitelisted crawlers list includes Google's domains in its example. A greylisted IP that is not Google's, or a custom rule, can still affect requests that only claim to be Googlebot.

Can I whitelist an AI crawler myself on shared hosting?

Usually not on the server firewall, which is an administrator control. On WordPress with the Imunify Security plugin, a site administrator can add IPs, ranges or user agents to the AI Bot Management allowlist, which skips its bot checks and rate limits only.

Why does curl get a 415 from my site?

For greylisted IPs, Imunify360 returns 415 Unsupported Media Type when the Accept header does not start with text/, and curl sends */* by default. Repeat the request with -H 'Accept: text/html' to see the challenge page instead.

Does robots.txt help with an Imunify360 challenge?

No. WebShield answers before the request reaches your site, so a crawler that is challenged never gets your page, whatever robots.txt allows.

Check what crawlers receive

After your host makes a change, the free check fetches one page the way AI crawlers do, with no account needed. It runs from Crawlito's servers, so an IP-based rule may treat it differently from OpenAI's or Anthropic's network, and your access log remains the evidence for the real crawlers.

Filed under CDN and bot protection. Topics: bot-protection, waf, ai-crawlers, troubleshooting