Skip to content

Akamai Bot Manager blocking AI crawlers: how to find and fix it

How Akamai Bot Manager classifies AI crawlers as known bots, why an AI search crawler can be denied, and how to find and fix it without opening the door to scrapers.

By Crawlito · Updated 7 min read

In short

  • Akamai keeps a directory of known bots, organised into categories, and Bot Manager applies the action you set for each category, such as monitor, allow, deny, tarpit or challenge.
  • Akamai has an AI bots category in its directory, and a single deny on that category blocks AI search crawlers and user fetchers along with training crawlers.
  • Akamai's own guidance is to run Bot Manager in monitor mode first and to add a robots.txt rule for known bots before you deny them.
  • A curl request with a crawler's user-agent from your own machine is not a verified known bot, so its result can differ from what the real crawler gets.
  • The fix is a per-bot or per-category action change in your security configuration, checked against the operator's published IP ranges, not a blanket allow on a user-agent string.
On this page

If an AI search crawler gets 403 errors or an "Access Denied" page from a site on Akamai, the usual cause is a Bot Manager action: the crawler is recognised as a known bot, its category is set to deny or challenge, and the request never reaches your origin. You fix it by changing the action for that bot or category in your security configuration, after confirming which bot is affected and what you want it to do.

This guide covers how Akamai classifies bots according to its documentation, why AI crawlers end up blocked, how to find the evidence, and how to fix it narrowly.

How Akamai classifies bots

Akamai's detection methods documentation describes several layers. The ones that matter for AI crawlers are:

  • Known bots. Akamai maintains a directory of bots it has identified and validates them, so you can handle each type differently. Its November 2024 post on managing AI bots says the directory contains only multi-factor definitions, not just a user-agent match, and that AI crawlers such as OpenAI's are already in it.
  • Categories. Known bots are grouped into categories, called Akamai-categorized bots in the documentation. Akamai says it has a category for AI bots. Category names and contents change as Akamai updates the directory, so read the list in your own Control Center rather than relying on any article, this one included.
  • Custom bots. You can define your own bot categories and bots inside them, for example to mark a partner's or your own tool as friendly.
  • Transparent and active detection. Requests that do not identify themselves are scored on signals such as header patterns and browser mismatches, or tested with an interaction. These catch scrapers that pretend to be browsers.

Bot Manager then applies the action you configured for each category or detection. The actions Akamai's adversarial bots guide names include monitor, allow, deny, tarpit and challenge. Akamai also packages basic bot visibility and controls in App & API Protector, with more detection in Bot Manager, so the exact options depend on your contract.

Why AI search crawlers end up denied

AI crawlers are not one thing. There are training crawlers such as GPTBot and ClaudeBot, search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, and user-triggered fetchers such as ChatGPT-User and Perplexity-User. The training vs search vs user crawlers guide explains what blocking each type costs you.

Three configurations block the ones you probably wanted:

  1. A deny on a broad AI category. Someone set the AI bots category to deny to stop training crawlers. If the category also contains search crawlers and user fetchers, they are denied too. Akamai's own 2024 post warns that a single AI category can pose challenges and suggests acting on individual bots instead.
  2. A challenge action on a category of bots. A crawler does not solve a challenge. It receives the challenge response and records your page as unreadable. The post on WAF challenge pages shows what that response looks like.
  3. A bot that is not in the directory yet. A newly launched fetcher with a new user-agent is not a known bot. It is then judged by transparent detection like any other unidentified client, and a strict conditional action can block it.

Not every block is Bot Manager. Rate controls, IP or geo rules in your Web Application Firewall, and origin-level rules can produce the same 403. Find which layer acted before you change anything.

How to find the block

Step 1: confirm the crawler is getting errors

Start from your origin or CDN logs, filtered by user-agent, and compare status codes over time:

bash
grep -E "OAI-SearchBot|Claude-SearchBot|PerplexityBot|ChatGPT-User|Perplexity-User" access.log \
  | awk '{print $9}' | sort | uniq -c

If Akamai blocked the request at the edge, your origin never sees it. In that case a drop in crawler hits at the origin is the clue, and the evidence is in Akamai's logs and reports, not in yours. The guide to AI crawlers in server logs covers where to look for each crawler.

Step 2: read Bot Manager's view

Bot Manager's reports show which bots and categories hit which pages and what action was taken. Look for the crawler by name, check its category, and note the action. If you export security logs from Akamai to a log platform, filter on the bot name or category and the action.

Step 3: reproduce carefully

You can request a page with a crawler's user-agent:

bash
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  https://www.example.com/

Read the result with care. Akamai validates known bots with more than the user-agent string, so a request from your laptop with OAI-SearchBot's string is not the verified OAI-SearchBot. It may be treated as an impostor, or scored like any other client. A 403 here tells you an impostor is blocked, which may be exactly what you want. It does not prove the real crawler is blocked. Bot Manager's reports for the real crawler are the evidence that counts.

When a request is denied by Akamai's edge, the response is often a short "Access Denied" HTML page with a reference number. Save that number: the site's Akamai administrators can translate it in Edge Diagnostics (Translate Error String) or Security Center to find the exact request and the protection that denied it. See how to trace an Akamai reference number.

How to fix it without letting scrapers in

Work from the narrowest change outward.

Set actions per bot or per purpose

Decide what you want for each crawler group, then express it in the security configuration:

Crawler groupExamplesTypical action for a site that wants AI search visibility
AI search crawlersOAI-SearchBot, Claude-SearchBot, PerplexityBotAllow or monitor
User-triggered fetchersChatGPT-User, Claude-User, Perplexity-UserAllow or monitor
Training crawlersGPTBot, ClaudeBot, CCBotYour policy: monitor, or deny if you opt out of training
Unidentified clients posing as browsersScrapersConditional actions from detection methods

If your Akamai categories mix these groups, set actions on the individual bots that need different treatment, or move them into a custom category with its own action.

Use robots.txt for policy, Bot Manager for enforcement

Akamai's detection methods page notes that known bots usually honor robots.txt and recommends adding a robots.txt entry before resorting to a deny. That is a cleaner way to opt out of training: a Disallow: / for GPTBot or ClaudeBot is a stated preference those operators document that they respect, and it does not affect OAI-SearchBot or Claude-SearchBot. Keep deny actions for bots that ignore robots.txt.

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow: /account/

Do not allowlist a user-agent string

A rule that allows any request whose user-agent contains PerplexityBot lets in every scraper that copies that string. Rely on Akamai's known-bot validation, or, for a custom rule, match on the operator's published IP ranges. The guide to verifying real AI crawlers lists the IP sources per operator, and the planned post on AI crawler IP ranges for firewall allowlists covers keeping them current.

Start in monitor mode, then tighten

Akamai's adversarial bots guide recommends implementing Bot Manager in monitor mode first and watching for a few weeks before switching to mitigating actions such as deny, tarpit or challenge. If you are changing AI crawler actions, do the same: set the categories you are unsure about to monitor, review what they request, then decide.

Recheck after the change

After a configuration change is activated, watch for:

  1. Crawler requests reaching your origin again with 200 responses.
  2. No new denies for that bot in Bot Manager's reports.
  3. The page content, not a challenge, in the responses your logs record.

Changes on other platforms follow the same pattern. The AWS WAF Bot Control guide shows the equivalent fix in AWS.

Frequently asked questions

Does Akamai block AI crawlers by default?

Akamai does not publish a single default for every customer. Actions are set per security configuration, so check the actions on your bot categories and detections in your own Control Center.

Can I allow OAI-SearchBot but deny GPTBot on Akamai?

Yes, if they are handled separately. Set actions per bot, or put them in custom categories with different actions. Disallowing GPTBot in robots.txt achieves the same opt-out without a deny.

Why does my curl test with a crawler user-agent get blocked?

Because it is not the real crawler. Akamai validates known bots with more than the user-agent string, so a request from your IP address with a copied string can be treated as an impostor.

Will tarpit or challenge actions stop crawlers?

Yes, in practice. A crawler does not solve challenges, and a tarpit slows or holds the request, so either action can leave the crawler without your content.

Where do I see which action Akamai took?

In Bot Manager's reports and in the security event logs for your configuration. Keep the reference number from any "Access Denied" page you see while testing.

Check what an AI crawler receives

Edge blocks are hard to see from inside your own network. The free check shows what one page returns to AI crawler requests. For a full picture across templates, with evidence from your logs, an audit includes the configuration changes to make.

Filed under CDN and bot protection. Topics: bot-protection, waf, ai-crawlers, server-logs, troubleshooting