Vercel AI Bots ruleset: block training, allow AI search
Vercel block AI bots without losing AI search: set the AI Bots ruleset per category, keep Bot Protection from challenging crawlers, and check the result.
By Crawlito · 9 min read
In short
- Vercel's AI Bots managed ruleset has two categories, AI Crawlers and AI Assistants, and you can set each one to log or deny separately; it is available on all plans and inactive by default.
- In Vercel's verified bots directory, GPTBot and ClaudeBot are AI crawlers, while OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-triggered fetchers are AI assistants, so denying only AI Crawlers blocks those training bots and leaves those search bots alone.
- The categories are not a clean training-versus-search split: the same directory lists Applebot and Meta-WebIndexer as AI crawlers, so check bots.fyi before you deny a whole category.
- Bot Protection in challenge mode serves a JavaScript challenge that only a real browser can pass; it skips verified bots, but a crawler Vercel cannot verify gets the challenge instead of your page.
- Vercel runs custom rules before managed rulesets, so a custom deny or challenge rule beats whatever you set in AI Bots, and a custom bypass rule is how you let a specific bot through.
On this page
- What does the AI Bots managed ruleset do?
- AI Crawlers vs AI Assistants: how do Vercel's categories map to training and search?
- How to set Log or Deny per category
- Why can Bot Protection hide your pages from crawlers?
- How do custom rules and bypass decide the result?
- How to verify what a crawler receives after you publish
- Frequently asked questions
- Check what your Vercel site returns
To block AI training crawlers on Vercel and keep AI search, open the project's Firewall rules, set the AI Bots managed ruleset to Deny for AI Crawlers, leave AI Assistants on Allow (or Log), and publish. Keep Bot Protection in Log mode unless you have tested it, because its challenge page is the most likely way in this setup to hide your content from crawlers you wanted.
Two honest caveats first. Vercel's categories are its own, and they do not map perfectly onto "training" and "search". And this controls what Vercel's firewall does with requests; it does not decide whether an assistant cites your site. At the time of writing, everything below matches Vercel's documentation, so check the linked pages before you rely on an exact label or command.
What does the AI Bots managed ruleset do?
It lets you log or deny traffic from AI bots that Vercel recognizes. Per Vercel's Bot Management docs, the ruleset covers bots that crawl for training data, for search, or for user-triggered fetches, and Vercel maintains the list of known bots itself. When a new AI bot is added to the list, it is handled according to the action you already set.
Facts worth knowing before you touch it:
- It is available on all plans.
- It is inactive by default. In the dashboard that state is labeled Allow, and matching traffic is not evaluated.
- A Deny returns a
403 Forbidden, and per the firewall concepts page the request does not reach your application. - Log records the traffic and lets it through, which makes it the safe first step.
Vercel's managed rulesets page says you can control the two categories independently. That is the feature that makes "block training, allow search" possible in one ruleset.
AI Crawlers vs AI Assistants: how do Vercel's categories map to training and search?
Vercel's documentation defines the two categories like this:
- AI Crawlers: bots that crawl and collect content for training.
- AI Assistants: bots that retrieve pages in response to user actions or for search queries.
Vercel's October 2026 changelog entry words the first one a little wider, as bots that collect content for search indexing and model training. So read the categories as a guide, and look at where the specific bots you care about sit. The verified bots directory on the Bot Management page labels them like this:
| Bot | Vercel label | Purpose (from the directory descriptions) |
|---|---|---|
| GPTBot | ai_crawler | Crawls content to improve OpenAI's models |
| ClaudeBot | ai_crawler | Collects content that could contribute to training |
| CCBot | ai_crawler | Common Crawl, for AI training and research |
| OAI-SearchBot | ai_assistant | Indexes sites for ChatGPT search |
| Claude-SearchBot | ai_assistant | Improves Claude search results |
| PerplexityBot | ai_assistant | Indexes sites for Perplexity search |
| ChatGPT-User, Claude-User, Perplexity-User | ai_assistant | Fetch a page because a user asked |
| Applebot | ai_crawler | Powers Spotlight, Siri and Safari search, and may train Apple's models |
| Meta-WebIndexer | ai_crawler | Provides search results for Meta AI |
| Googlebot, Bingbot | search_engine_crawler | Search engines, not in the two AI categories |
Two things follow. First, denying AI Crawlers while leaving AI Assistants on Allow does what you want for OpenAI, Anthropic and Perplexity: GPTBot and ClaudeBot are blocked, and the search and user-triggered bots are not. OpenAI says its GPTBot and OAI-SearchBot settings are independent, and that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers (they can still appear as navigational links), so the assistant label is the one you want left alone.
Second, the category is wider than "training". A blanket deny on AI Crawlers also catches Applebot and Meta-WebIndexer, which that directory describes as feeding search features. bots.fyi, the catalog Vercel points to for the full classification, can tag the same bot differently from the docs table, so confirm there. For the general split behind all of this, see AI training crawlers vs search crawlers.
Google-Extended needs separate handling. Vercel's directory and bots.fyi list a google-extended entry as an AI crawler, but Google says Google-Extended has no separate HTTP user agent string and does not affect inclusion in Google Search. Google describes it as a robots.txt control token, with crawling done under its existing user agent strings, so a user-agent rule cannot target it. Set it in robots.txt. The robots.txt version of this policy covers it.
How to set Log or Deny per category
In the dashboard
- Open your project, then Firewall, then Rules.
- In the Bot Management section, open the configure icon for AI Bots.
- Set an action for AI Crawlers and a separate action for AI Assistants. The managed rulesets page lists Log and Deny.
- Choose Review Changes, check the diff, and select Publish.
Start with Log on both categories, look at the traffic for a day, then move AI Crawlers to Deny.
With the CLI
Vercel's CLI reference shows ai-bots as a managed rule with the actions allow, log, challenge and deny. This example denies AI Crawlers and keeps AI Assistants allowed:
vercel firewall rules update ai-bots --category ai-crawlers --action deny --yes
vercel firewall rules update ai-bots --category ai-assistants --action allow --yes
vercel firewall diff
vercel firewall publishChanges to managed bot rules are staged as drafts until you publish. vercel firewall publish shows the staged changes and asks you to confirm; in a script, pass --confirm-name with the project name instead. One thing to watch: the October changelog shows vercel firewall rules edit ai-bots ..., while the current CLI reference documents rules update. Use the one your installed CLI accepts, and avoid challenge for either category: the managed rulesets page lists only allow, log and deny for each category, and a crawler cannot solve a challenge, so it will not reach your content.
Why can Bot Protection hide your pages from crawlers?
The AI Bots ruleset blocks on purpose. Bot Protection is the other managed ruleset, and it can block by accident.
Per Vercel, Bot Protection is available on all plans and inactive by default. In challenge mode it serves a JavaScript challenge to traffic that is unlikely to be a browser, including requests that falsely claim to be one. A challenge works by running code in the browser, so Vercel's own docs say non-browser clients such as bots and scripts cannot pass it. A client that passes gets a session valid for one hour.
Crawlers that fetch raw HTML do not run JavaScript, so they never get past the challenge, and what they receive is the checkpoint page instead of your content. The general mechanism is in why a WAF challenge page shows AI bots nothing.
The protection for crawlers is verification. Bot Protection automatically excludes verified bots, which Vercel recognizes by IP address, reverse DNS or cryptographic signatures. The directory includes OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot. That leaves three ways to get challenged anyway:
- The crawler is not on the verified list. Vercel's guide says bots that cannot be verified do not bypass protections. A newer or smaller AI crawler, or an audit tool, sees what an unverified client sees.
- Someone is spoofing or testing. A
curlrequest with a crawler's user agent comes from your IP, not the operator's, so it can be challenged even though the real crawler is not. - A reverse proxy sits in front of Vercel. Vercel says Bot Protection does not work with a proxy such as Cloudflare in front, because the proxy masks the signals it relies on.
The fix is to start in Log mode, which Vercel's bot management guide describes as blocking no requests, then read the logged traffic before moving to Challenge. If a crawler you want is getting caught, use a bypass rule, covered next.
How do custom rules and bypass decide the result?
Order decides it. Vercel's managed rulesets page says the WAF runs custom rules first and managed rulesets second. The CLI's vercel firewall status shows the full evaluation order: bypass, system mitigations, attack challenge mode, IP blocks, custom rules, then the Bot Protection, AI Bots and OWASP managed rulesets.
Two consequences:
- A custom deny or challenge rule wins. Custom rules stop execution when they deny or challenge a request. If you wrote a rule that blocks anything with
botin the user agent, it stops OAI-SearchBot no matter what the AI Bots ruleset says. - Bypass is the way to let a bot through. Vercel documents bypassing rulesets with a custom rule whose action is bypass, and says to put it above any blocking custom rule.
A bypass for one search crawler looks like this:
vercel firewall rules add "Allow OAI-SearchBot" \
--condition '{"type":"user_agent","op":"sub","value":"OAI-SearchBot"}' \
--action bypass --yes
vercel firewall rules move "Allow OAI-SearchBot" --first --yes
vercel firewall publishBe careful with that rule. A user agent is only a header, so anyone can send it, and a matching request skips the rules after it. Vercel's guide warns that spoofed metadata is possible and suggests keeping rate limits or other controls. Pair the user agent with the operator's published IP ranges (OpenAI, for one, recommends allowing its ranges), or use verified-bot checks before you decide a request is real. Also note that rules in vercel.json support only challenge and deny, so a bypass has to be set in the dashboard or CLI.
How to verify what a crawler receives after you publish
Check three things, in this order.
- Confirm the configuration.
vercel firewall bot-managementprints the current action for Bot Protection and AI Bots, andvercel firewall statusprints the evaluation order. - Read the traffic. On the Firewall page, the live view covers a 10-minute window and the Traffic tab groups requests by user agent and filters by action. In the CLI,
vercel firewall traffic list --action deny --dimension user-agentshows which user agents were denied. Every plan retains the last 24 hours; going back seven days needs Observability Plus. - Request the page as each bot. Compare the status and the body.
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.3" https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot/1.4" https://example.com/With AI Crawlers on Deny, you expect the first to return 403 and the second to return 200. Read that result with one limit in mind: Vercel does not document whether the AI Bots ruleset matches on the user agent string alone or on verified identity, and your request comes from your own IP. If the curl result and the live firewall traffic disagree, trust the traffic. To look for a challenge, request a page with a Chrome-like user agent and a bot user agent, then compare the body with a sentence you know is on the real page. The curl crawler test guide has the full routine.
If the pages come back with a 200 but still look empty to a crawler, the problem is rendering, not the firewall. See Next.js SEO for AI crawlers.
Frequently asked questions
Does denying AI Crawlers remove my site from ChatGPT search?
Not by Vercel's classification: GPTBot is an AI crawler and OAI-SearchBot is an AI assistant. OpenAI says the two settings are independent and that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. Confirm both labels on bots.fyi, since categories can change.
Do the AI Bots ruleset and robots.txt do the same job?
No. The ruleset blocks requests at the firewall, while robots.txt asks crawlers to follow rules. Vercel's docs do not describe how the two interact. Keep robots.txt for the bots that respect it and for tokens like Google-Extended that have no user agent string of their own.
Do I need a paid plan?
Per Vercel's docs, the AI Bots ruleset, Bot Protection and custom rules are available on all plans. Longer firewall traffic history needs Observability Plus.
Should I switch Bot Protection to Challenge?
Only after a period in Log mode. A challenge cannot be solved by crawlers or scripts, so any unverified bot you wanted gets the challenge page. Use bypass rules for the exceptions.
Does a firewall change need a redeploy?
No. Vercel says applying a custom rule takes effect immediately without redeployment. You do still have to publish the staged changes.
Check what your Vercel site returns
After you publish, run the free check on a page. It reads the page from Crawlito's servers, which are not on any operator's published IP ranges, so it shows what an unverified client receives, including a challenge page if one is served. For a whole site, an audit covers up to 500 pages with evidence and prioritized fixes.
Filed under CDN and bot protection. Topics: bot-protection, waf, ai-crawlers, nextjs, troubleshooting