Skip to content

DataDome and AI crawlers: default blocks and verified bots

Which AI crawlers DataDome blocks by default, how its Verified Bots and AI agent policies work, and how to allow the crawlers you want without trusting user agents.

By Crawlito · 8 min read

In short

  • DataDome's documentation says automated traffic from bots or AI agents it cannot authenticate is classified as Threat Detection and blocked by default.
  • DataDome applies recommended policies to known AI agents automatically, so whether a given AI crawler is allowed depends on DataDome's recommendation unless you set your own policy.
  • A bot joins DataDome's Verified Bot category with a dedicated user agent plus one of five authentication methods: Web Bot Auth, reverse DNS, static IP, a dynamic IP list or a private AS.
  • The Allow policy is not supported for AI Threats Detection models, and DataDome says custom rules override AI Threats Detection policies, so a custom rule is the documented way to let a crawler through.
  • DataDome's Explore view shows the user agent, rule name and response for each request, which is the evidence for what a named crawler actually received.
On this page

DataDome blocks many AI crawlers by default. Its documentation says traffic from bots or AI agents it cannot authenticate is treated as a threat and blocked, and known AI agents get a recommended policy from DataDome unless you set your own. To let in the AI crawlers you want, you set an Allow policy for the crawler's AI agent, or write a custom rule that matches the crawler's verified model or its real origin. You then confirm in DataDome's Explore view what the crawler received.

This guide is for ecommerce and publisher teams on DataDome. It covers how DataDome identifies bots, what it does with AI crawlers out of the box, where the controls are, and how to check a named crawler. For the signing protocol itself, see Web Bot Auth explained. For the same problem on another vendor, see Akamai Bot Manager and AI crawlers.

What DataDome does with AI crawlers by default

DataDome's documentation describes two default positions, and they depend on whether it can verify where a request comes from.

Unverified bots are blocked. DataDome's bot authentication page says automated traffic from unauthenticated bots or AI agents is classified as "Threat Detection" and blocked by default. It also says DataDome classifies and authenticates thousands of commercial bots and AI agents. A crawler that DataDome cannot authenticate, or a scraper that only copies a crawler's user agent, falls into that blocked group.

Known AI agents get a recommended policy. The Agentic Trust access control page says DataDome ships recommended policies for all known AI agents, calibrated to each agent's business relevance and overall trust level. Its AI agent identification page adds that new catalog models are validated in simulation mode, then appear to customers deactivated. Customers are notified and get two weeks to review, after which the model is activated and the recommended policy applies to anyone without a custom policy.

DataDome has also published threat research saying that many LLM crawlers, OpenAI's GPTBot among them, are blocked by default and that customers can override the decision. Treat that as a snapshot from the time of writing. DataDome's product documentation does not publish a per-crawler table of recommended policies, and recommendations can change when a new model is added.

The practical reading: do not assume a crawler is allowed or blocked on your account. Read the policy for each crawler you care about in your own dashboard.

How DataDome verifies a bot

A bot operator who wants to be in DataDome's Verified Bot category, or have an AI agent identified, needs a dedicated user agent and one of five authentication methods, per the bot authentication page:

MethodHow DataDome checks it
Web Bot AuthVerifies HTTP message signatures (RFC 9421) cryptographically
Reverse DNSTwo-way reverse DNS and forward DNS lookup, the method Google and Bing use
Static IPThe operator gives DataDome the full list of possible IPs
Dynamic IP listDataDome downloads an IP list the operator maintains as CSV, JSON or plain text
Private ASA private corporate autonomous system, not a common hosting AS such as AWS, GCP or Hetzner

The operator then submits a verification request through DataDome's form; Web Bot Auth requests use a separate form. DataDome's identification page rates identification methods by strength: Web Bot Authentication, cryptographic token verification, trusted IP source lists and reverse DNS are "high", network attribution and client-side fingerprinting are "medium", and server-side fingerprinting is "low". User agent matching on its own is not listed as an identification method.

Identity is not the same as permission. The Agentic Trust page offers an Intent-Based policy that blocks only the requests DataDome's intent detection models judge fraudulent, so a verified agent can still be stopped for what it does.

The Verified Bots page groups verified bots into categories such as Major Search Engines, Alt Search Engines, SEO Software, Social Networks, Digital Library, Web Aggregator and Partners. The Agentic Trust catalog groups AI agents differently, into AI Crawler, AI Assistant, Autonomous Agent and Agentic Browser.

Where the controls live

DataDome has two places where an AI crawler's fate is decided, and it helps to know which one you are looking at.

Bot Protect access control

DataDome's Manage DataDome Protection documentation covers four areas: AI Threats Detection, Verified Bots, Custom Rules and Rule Responses. The responses page lists what a rule can do to matching traffic:

ResponseEffect on a crawler
AllowRequest reaches your site
TimeboxingAllowed only in chosen hours on chosen days, otherwise shown a Captcha, blocked or sent to Device Check
Rate LimitingAllowed up to a threshold per hour or day, then shown a Captcha, blocked or sent to Device Check
CaptchaShown a challenge instead of the page
Device CheckRedirected to an interstitial page that checks the device
BlockFully blocked from the endpoint
CustomAllowed, but tagged with an HTTP header your backend must act on
MonetizeHanded to a monetization partner's workflow, which can redirect, allow or block

Captcha and Device Check are effectively blocks for a crawler. It receives the challenge page instead of your content, as explained in WAF challenge pages and AI crawlers.

The AI agents dashboard

The Agentic Trust access control page sets policies per agent at three levels: a default policy for all of an agent's traffic, endpoint-level policies in the agent's detail view, and custom rules inside that view. The documented policies are Intent-Based, Allow, Block, Rate Limit, Time-Box and Monetize. The page does not say how the three levels interact when more than one matches, so test any combination you rely on.

How to allow the AI crawlers you want

Decide first which crawlers you want. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, and user-triggered fetchers such as ChatGPT-User and Perplexity-User, fetch pages for AI answers. Training crawlers such as GPTBot and ClaudeBot collect content for model training. The training vs search vs user crawlers guide covers the trade-off.

If the crawler is in the AI agents catalog

Open the agent in the AI agents dashboard and set its default policy to Allow, or to Rate Limit or Time-Box if load is the concern. Endpoint-level policies let you allow some endpoints and block others; DataDome's example is an agent allowed to browse products and complete purchases but blocked from creating accounts.

If the crawler is caught by Threats Detection

This is the common trap. DataDome's Threats Detection page says the only policies for AI Threats Detection models are Device Check, Slider and Block. Allow is not supported, and for some models the policy cannot be changed at all. The page suggests deactivating a model instead, but that switches it off for all traffic it would catch, not only the crawler.

The narrower option is a custom rule. The custom rules page says custom rules override AI Threats Detection policies and that the higher-priority rule applies first when two rules match.

The steps documented there: go to Access Control > Custom Rules, click Create a Custom Rule, enter a query, select the endpoints, optionally choose a verified bot or AI agent from the Model dropdown, click Check Rule, then Add Rule with a response, a name and a priority. Only Admin and Editor users can add rules.

Match on the verified model or on origin, never the user agent alone

Where DataDome already has a verified model for the crawler, selecting it in the Model dropdown is the better option, because DataDome maintains the verification. Otherwise, match on origin. A rule that allows any request whose user agent contains a crawler's name lets every scraper that copies the string through. DataDome's query syntax supports CIDR ranges and a reversedns field, so you can combine the user agent with the operator's published ranges:

useragent:*PerplexityBot* AND (ip:192.0.2.0/24 OR ip:198.51.100.0/24)

The ranges above are documentation placeholders. Take the real ones from the operator's own list, as described in how to verify real AI crawlers, and remember that a hard-coded list goes stale when the operator adds addresses. Field names and values in DataDome queries are case sensitive, and the * wildcard cannot be used inside double quotes.

How to check what a named crawler receives

A test from your own laptop with a crawler's user agent tells you little. DataDome authenticates crawlers by signature, DNS or IP, so your request is not the verified crawler and may be blocked as an impostor. The evidence is in DataDome's own logs.

  1. Open Explore. The Explore page filters by time range, domain and endpoint, by traffic type (AI Threat Detection, Verified Bots, Custom Rules, Human and Uncategorized) and by model. The Agentic Trust Explore view also has an Agent quick filter for a single AI agent.
  2. Search for the crawler. If it has no model yet, match the user agent, for example useragent:*OAI-SearchBot*, and add responsestatus or reversedns to narrow it.
  3. Read the rule name and response. The list view shows date, URL, user agent, rule name, IP and response for each request, and the request details show DataDome's detection and the response code.
  4. Compare with your origin. A request DataDome challenges or blocks never reaches your server. In the Protection API reference, the module returns any API status other than 200 straight to the client and does not contact your backend. A drop in a crawler's hits in your origin logs, with a matching rule name in Explore, confirms the block.

After you change a policy or add a rule, repeat the search a day later. You want the crawler's requests showing your Allow rule or model and an allowed response, not a challenge.

Frequently asked questions

Does DataDome block GPTBot by default?

DataDome's threat research has said GPTBot is blocked by default and that customers can override the decision. Recommended policies can change, so check the policy shown for each OpenAI agent in your own dashboard.

Why can't I set Allow on a DataDome threat detection model?

DataDome's documentation says Allow is not supported for Threats Detection models, which offer only Device Check, Slider and Block. Create a custom rule instead, since custom rules override AI Threats Detection policies.

Is it safe to allow an AI crawler by user agent in DataDome?

No. Anyone can send a crawler's user agent. Use the crawler's verified model, or combine the user agent with the operator's published IP ranges or reverse DNS in a custom rule.

Does DataDome support Web Bot Auth?

Yes. DataDome lists Web Bot Auth as one of its five bot authentication methods and rates it a high-confidence identification method.

Check what AI crawlers get from your site

DataDome's settings are only half the picture: robots.txt, rendering and other layers can still stop a crawler you allowed. Run the free check on an important page to see what the main AI crawlers can fetch. If you want every crawler's access reviewed across your templates with evidence, an audit covers edge settings and robots.txt together.

Filed under CDN and bot protection. Topics: bot-protection, ai-crawlers, web-bot-auth, waf