Skip to content

Imperva WAF and AI crawlers: Client Classification settings

How Imperva Cloud WAF classifies AI crawlers as AIBot clients, how Good Bots and Bad Bots settings decide access, and how to trace a block by incident ID.

By Crawlito · 9 min read

In short

  • Imperva Cloud WAF classifies every visitor with a client type and a client ID, and at the time of writing its list puts GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User and Perplexity-User all in one client type, AIBot.
  • Imperva defines AIBot as a bot used for LLM training, so a custom rule that blocks ClientType == AIBot also blocks AI search crawlers and user-triggered fetchers.
  • Bot Access Control has three options: allow Good Bots, block Bad Bots, and challenge suspected bots that Imperva cannot classify.
  • A block from Bot Access Control shows error code 16 on the Imperva error page, while code 12 means the request got a CAPTCHA challenge instead of content.
  • The incident ID on an Imperva block page has the format Session ID-Request ID, and filtering Security Events by it shows the client app, the action and the violation behind the block.
On this page

Imperva Cloud WAF decides what an AI crawler gets from its own client classification: every request is matched to a named client, such as GPTBot or OAI-SearchBot, and a client type, such as AIBot or SearchBot. Bot Access Control then allows good bots, blocks bad bots and challenges bots it cannot classify, and your custom rules can act on the client type or ID. The catch for AI search is that Imperva puts training crawlers, search crawlers and user-triggered fetchers in the same AIBot type, so one broad rule can block all of them.

This post covers how Imperva classifies AI crawlers, which settings decide whether they get through, and how to trace a block from the incident ID on the error page to the event that caused it. Imperva's documentation now lives on the Thales documentation portal, and every setting below comes from those pages at the time of writing.

How Imperva classifies AI crawlers

Imperva's Client Classification page lists the client types it assigns and close to a thousand named clients, each with a numeric client ID. The types that matter here:

Client typeImperva's descriptionExamples from Imperva's list
AIBotA bot used by generative AI for LLM trainingGPTBot, ClaudeBot, CCBot, OAI-SearchBot, ChatGPT-User
SearchBotA bot that indexes data for a search engineGooglebot, MSN/Bing Bot, DuckDuckBot
CrawlerKnown crawling bots that harvest data for their own purposesAmazonBot, AhrefsBot, Barkrowler
HackingToolProgrammable tools and libraries, malicious or notcURL, python-requests, Headless Chrome
UnknownBots that fit no classification or whose purpose is unknownGeneric "Bot" entries

The AIBot description says training, but the list itself is broader. At the time of writing these AI clients all carry the AIBot type:

Client IDClient nameWhat the operator uses it for
1020GPTBotTraining
1019OAI-SearchBotSearch
987ChatGPT-UserUser-triggered fetches
998ClaudeBotTraining
1023Claude-SearchBotSearch
1022Claude-UserUser-triggered fetches
1003Perplexity AIListed under this name
1021Perplexity-UserUser-triggered fetches
96CCBotTraining (Common Crawl)
1044Meta-ExternalAgentAI crawler

The purposes in the right-hand column come from each operator's documentation, not from Imperva. The training vs search vs user crawlers guide explains why the difference decides what blocking costs you. Imperva keeps adding clients, so read the current list or pull it from the API rather than copying IDs from any article, this one included.

Two entries are worth knowing when you read events: Imperva has clients named "Google Bot Impersonator" (112) and "Good Bot Impersonator" (1000), both typed as Crawler. The list gives no description for them, but the names suggest where a request lands when it claims to be a known crawler and Imperva does not accept the claim.

Not every AI-related client is AIBot, either. In the same list, Meta-ExternalFetcher (1045) is typed Crawler, while Apple Bot (617) is typed AIBot even though the Crawler type description names Applebot as an example. Check each client you care about rather than reasoning from the type.

Bot Access Control: Good Bots, Bad Bots and suspected bots

The bot settings are under Application > Websites > (your site) > Website Settings > Security. Imperva's Security Settings page describes three options:

  1. All Good Bots (like Google and Pingdom) will be allowed to access your site. Good bots are allowed by default. The Good Bots link opens a list where each bot has a checkmark, which means it is not blocked. Imperva notes that good bot requests still pass through the WAF rules.
  2. Block Bad Bots (like comment spammers and scanners). Bad bots, such as vulnerability scanners and DDoS bots, are denied by default. You can add more bots with Block additional bad bots, or allow one with Add exception using its Client app ID.
  3. Require all other suspected bots to pass additional challenges. A bot Imperva cannot classify is a Suspected Bot. With this option on, those clients must complete a CAPTCHA or other challenge.

The same page says Googlebot and all other search engine bots are marked as good bots. Imperva's Bot Mitigation page adds that unidentified bots, those not listed in the client database, are treated as suspicious by default.

Imperva does not document which AIBot clients are on the Good Bots list for a new site, or what action applies to a good bot whose checkmark you remove. Open the list in your own console and look for each AI crawler by name. If a search crawler or user fetcher you want is unchecked, that is the first thing to fix.

Option 3 matters for new AI fetchers. An operator that launches a new user agent is not in the client database until Imperva adds it, so with suspected bot challenges on it can be sent a challenge it cannot pass. The WAF challenge pages guide explains why a crawler cannot solve one.

Custom rules that catch AI search by accident

Imperva rules can filter on the classified client, using the parameters on its Rule Filter Parameters page:

ParameterWhat it matches
ClientTypeThe client type Imperva assigned, such as AIBot
ClientIdThe client ID Imperva assigned, such as 1019
DecClientTypeThe client type declared in the User-Agent header
DecClientIdThe client ID declared in the User-Agent header

A rule written to stop training crawlers often looks like this, with the Block Request action:

ClientType == AIBot

Because OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User and Perplexity-User share that type, this rule blocks them too. If your policy is to opt out of training but stay reachable for AI search, list the exceptions:

ClientType == AIBot & ClientId != 1019 & ClientId != 987 & ClientId != 1023 & ClientId != 1022 & ClientId != 1021

Check the IDs against the current Client Classification list before you save. Imperva's Create Rules page describes the Alert action as a non-blocking alert useful for testing new rules, so run it as Alert first and see what it would have matched.

Prefer ClientId and ClientType over the declared versions. The declared parameters trust the user agent, so a rule on DecClientId treats a scraper with a copied string as the crawler it claims to be. For an opt-out from training, robots.txt is the cleaner tool, as allow AI search, block AI training shows, and it does not depend on any one WAF's grouping.

The challenge actions behave like a block for crawlers. Imperva says the Require Javascript Support and Require CAPTCHA Support tests are embedded in an HTML page, so a crawler that matches the rule gets the test and none of your content. Require Cookie Support can catch crawlers that do not keep cookies in the same way.

How to trace an Imperva block from the incident ID

When Imperva blocks a request, it shows an error page with an error code, a timestamp, the source IP, the Imperva proxy IP and ID, and an incident ID. The error codes page explains each code. These are the ones that touch crawlers:

CodeWhat it meansSIEM log entry
12The request got a CAPTCHA challenge page instead of the contentREQ_CHALLENGED_X
14Blocked by Block User or Block IP WAF settings, for the whole sessionREQ_BLOCKED_SESSION
15Blocked by WAF settings (Block Request, Block User or Block IP)REQ_BLOCKED_SECURITY
16Blocked by Bot Access Control or Block Specific SourcesREQ_BLOCKED_ACL
17Blocked by Block User WAF settings for a cookieless visitorREQ_BLOCKED_VISITOR

Many crawlers do not keep cookies, so watch for code 17 as well as 16. Error pages are HTML by default. If Enable content based error responses is turned on (under Origin and Network > General, Additional Settings), a request whose Accept header contains json and not html gets the same details as JSON, which is easier to capture from a script. A crawler that asks for HTML still gets the HTML page.

bash
curl -s -H "Accept: application/json" https://www.example.com/blocked-page
json
{
  "incidentId": "3411854340000000422-34793753560490",
  "errorCode": "16",
  "description": "This request was blocked by our security service"
}

The JSON is an illustrative response built from the field names and the sample incident ID on Imperva's Error Responses page, with the other fields trimmed. Then:

  1. In the Cloud Security Console, open Application > Security Events. Event data covers the last 90 days.
  2. Filter by the incident ID. Imperva's Security Events page says the incident ID has the format <Session ID>-<Request ID>, so the first part finds the session.
  3. Read Client Type and Client App for the session. They show how Imperva classified the request, for example AIBot and OAI-SearchBot, or Good Bot Impersonator.
  4. Expand the session and read Action, Status, Violations and Policy ID for the request. A custom rule shows a View rule action, and a Bad Bots violation shows Add exception to rule.

One caution from Imperva's documentation: the Bad Bots violation tag does not always mean the client was a bad bot. It can mean the client was unknown at first and was challenged, and Imperva calls the client shown in the session details the final, authoritative classification.

If you do not have a block page, work from the crawler side. Look for sessions whose Client App is an AI crawler you care about, use the quick filter (Filter by this value) on that field where it is offered, and look for Blocked or CAPTCHA actions. The Akamai version of this guide covers reading origin logs to spot crawler traffic that stopped arriving.

How to fix it narrowly

Match the fix to what the event showed:

  • Bot Access Control block (code 16) on a crawler you want. In the session's Set actions, choose Don't block this Bot, which adds it to the Good Bots list. Or check it on the Good Bots list in Security settings.
  • A custom rule matched. Narrow the filter with ClientId != exceptions, as above, or move training opt-outs to robots.txt.
  • A challenge (code 12) on an unclassified fetcher. You cannot pick a client that is not in the database, and Imperva notes that bots with the Unknown client type cannot be added as Bad Bots exceptions. Exceptions accept IPs, ranges and subnets, so add a narrow exception on the operator's published IP ranges. To get a client added to the Good Bots list, Imperva says to contact its support. The guide to verifying real AI crawlers lists where each operator publishes its IPs.
  • A session blocked by mistake. Release session in Security Events releases a session that was blocked unnecessarily. Change the setting or rule that caused it too, or the next session can be blocked the same way.

Avoid a user-agent exception for a crawler. Imperva's exceptions need an exact user-agent match, which breaks when an operator changes a version number, and anyone can send the same string.

Frequently asked questions

Does Imperva block GPTBot or ClaudeBot by default?

Imperva does not document a default for AIBot clients. They are listed by name in its client database, and Bot Access Control applies whatever your Good Bots and Bad Bots lists say, so check those lists for your site.

Why is OAI-SearchBot listed as an AIBot and not a SearchBot?

At the time of writing Imperva types the OpenAI, Anthropic and Perplexity clients as AIBot, including their search crawlers and user fetchers. Its SearchBot type holds traditional search engine crawlers such as Googlebot and Bingbot.

What does Imperva error code 16 mean?

The request was blocked by Bot Access Control or Block Specific Sources in the site's security settings. The SIEM log entry for it is REQ_BLOCKED_ACL.

Can I test an Imperva rule against a crawler with curl?

Only partly. Imperva keeps the declared client (from the user agent) separate from its own classification, which also looks at things like cookie and JavaScript support, so a curl request with a copied string may not be classified as the crawler. Security Events for the real crawler are the better evidence.

Check what AI crawlers receive from your site

After changing Imperva settings, run the free check on an important page to see whether the main AI crawlers can fetch it. If several sites or rule sets are involved, an audit reviews crawler access across up to 500 pages, with human review and a PDF of the evidence and prioritized fixes.

Filed under CDN and bot protection. Topics: bot-protection, waf, ai-crawlers, troubleshooting