Skip to content

Kinsta Bot Protection: what Block AI crawlers really blocks

Kinsta Bot Protection explained: the four protection levels, how verified bots are treated, and which AI crawlers the Block AI crawlers switch stops or leaves alone.

By Crawlito · 10 min read

In short

  • Kinsta Bot Protection was a beta feature in MyKinsta at the time of writing, with four levels: Block malicious traffic, Block automations, Challenge bots and Challenge everyone.
  • Kinsta says it allows most verified bots at every protection level and that Google and Bing crawlers are not affected, but AI crawlers and excessive-rate crawlers can be reclassified even when verified.
  • The Block AI crawlers switch blocks AI crawlers entirely, including verified ones, and Kinsta says it does not block search engine crawlers, AI search crawlers such as OAI-SearchBot and Claude-SearchBot, or user-initiated agents such as ChatGPT-User.
  • At Challenge bots and Challenge everyone, Kinsta automatically challenges excessive-rate AI crawlers, and a request that cannot complete a challenge is blocked.
  • To see what each crawler was allowed, challenged or blocked, use the Request breakdown chart and Analytics > Bot traffic in MyKinsta, not only the server access log.
On this page

Kinsta says its Bot Protection levels do not affect Google and Bing crawlers, and its Block AI crawlers switch is narrower than the name suggests. Kinsta documents that the switch blocks AI crawlers entirely, including verified ones such as GPTBot, but leaves search engine crawlers, AI search crawlers such as OAI-SearchBot and Claude-SearchBot, and user-initiated agents such as ChatGPT-User alone. The real risk for AI search is elsewhere: at the two challenge levels, an AI search crawler that Kinsta classifies as excessive-rate is challenged, and a request that cannot complete the challenge is blocked.

This post covers the MyKinsta screen itself: the levels, the switch, exceptions, and how to confirm what each crawler receives. For the WordPress settings underneath (robots.txt, SEO plugins, noindex), see the WordPress AI crawler guide. If you also run your own Cloudflare zone in front of Kinsta, the Cloudflare AI crawler settings guide covers that dashboard. Facts below come from Kinsta's Bot Protection documentation and changelog, read on 15 November 2026.

What Kinsta Bot Protection is

Bot Protection is a page in MyKinsta at Sites > sitename > Bot protection. Kinsta's changelog entry dated 28 May 2026 released the beta to everyone after a closed testing program, and the documentation still labels it beta at the time of writing. Kinsta's security page lists it as included in all plans.

Kinsta says it runs its own bot classification rules and complements them with Cloudflare's machine-learning bot score, which rates each visitor from 1 (highly likely automated) to 99 (most likely human). Its verified bots include those on Cloudflare's verified bot list. Kinsta sorts requests into these traffic types:

Traffic typeKinsta's definition
Verified botsLegitimate automation such as Google, Bing, monitoring and SEO tools, including AI bots on Cloudflare's verified list; Kinsta allows most of them at every level
Likely humansBot score 30 to 99
Likely botsUnverified automation, bot score 2 to 29
Automated trafficConfirmed automation, bot score 1; may include legitimate tools that are not verified
Unclassified trafficUnder 0.07% of traffic, usually harmless internal requests such as error pages
Malicious trafficAttack, DDoS and abuse traffic, blocked on every site
AI crawlersBots that collect content to be processed by AI systems
Excessive-rate AI crawlersAI crawlers sending unusually high request volumes

The important line for AI search is in the verified bots definition: Kinsta says AI crawlers and excessive-rate crawlers can be reclassified even if they are verified. Verified status keeps most crawlers clear of the level settings, but not of Kinsta's AI crawler rules.

Cloudflare's own verified bots page explains what "verified" means upstream: a bot must identify itself deterministically, through a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, and must behave reasonably, including obeying robots.txt and keeping request rates reasonable.

The four protection levels and what crawlers get

You change the level at Bot protection > Protection level > Change. For many sites at once, select them on the Sites page and use Actions > Bot protection > Change bot protection level.

LevelWhat Kinsta doesVerified crawlers
Block malicious traffic (default)Blocks malicious traffic, with DDoS mitigation and global IP and endpoint rulesMost allowed
Block automationsAlso blocks automated trafficMost allowed
Challenge botsBlocks automated and malicious traffic; challenges likely bots and unclassified trafficMost allowed; excessive-rate AI crawlers are challenged
Challenge everyoneBlocks automated and malicious traffic; challenges likely humans, likely bots and unclassified trafficMost allowed; excessive-rate AI crawlers are challenged

Three consequences follow from Kinsta's descriptions:

  • Google and Bing are not affected by the levels. Kinsta says widely recognized search engine bots, including Google and Bing, are on the verified bot directory and are not affected, while warning that lesser-known SEO crawlers that are not verified could be challenged.
  • Unverified crawlers can be blocked from Block automations upwards. A crawler that is not verified and scores as confirmed automation (bot score 1) falls into "automated traffic", which every level above the default blocks. Kinsta warns the same applies to unverified APIs, custom integrations, self-hosted monitors and deployment automation, and suggests the vendor register with Cloudflare's verified bots program.
  • A challenge usually works as a block for a crawler. Kinsta describes challenges as browser checks such as JavaScript tests, a CAPTCHA, or background validation, and says a request is blocked if the challenge fails or cannot be completed. It adds that the vast majority of automated requests cannot complete the verification, and WAF challenge pages explains why crawlers in particular fail them.

Kinsta's changelog compares Challenge everyone to breaking the glass in case of fire: a way to bring excessive traffic under control while you investigate, and unlikely to suit long-term use. If you raised the level during an attack or a scraping spike, put it back afterwards.

What the Block AI crawlers switch blocks and leaves alone

The switch lives at Bot protection > Block AI crawlers, or in bulk under Actions > Bot protection > Change AI crawler block. Kinsta's wording is that it "blocks these crawlers entirely, including verified ones", where an AI crawler is a bot that collects content to train, analyze or power AI models. GPTBot is the example Kinsta gives of a crawler that identifies itself.

Kinsta lists what the switch does not block:

Kept out of the switchExamples Kinsta names
Search engine crawlersGooglebot, Bingbot
AI search crawlersOAI-SearchBot, Claude-SearchBot
User-initiated AI agentsChatGPT-User

Kinsta also notes that Cloudflare categorizes Googlebot, Bingbot and Applebot as both search and training bots, so blocking training bots in Cloudflare's own dashboard can remove a site from search results. Kinsta's switch does not touch them.

Mapped against the crawlers people usually ask about, using each operator's stated purpose (see training, search and user crawlers):

CrawlerPurposeWith Block AI crawlers on
GPTBotOpenAI trainingBlocked (Kinsta's own example)
ClaudeBotAnthropic trainingNot named by Kinsta
CCBotCommon Crawl datasetNot named by Kinsta
OAI-SearchBotChatGPT search indexNot blocked
Claude-SearchBotClaude searchNot blocked
PerplexityBotPerplexity search indexNot named by Kinsta
ChatGPT-UserFetches a user asked forNot blocked
Googlebot, BingbotSearch indexingNot blocked

Kinsta does not publish the full list of crawlers the switch matches, so for anything it does not name, confirm in your own analytics, as shown below. Kinsta's documentation also says blocking AI crawlers "may still reduce how often your content is used or referenced by some AI tools", and advises monitoring your traffic after the change.

The switch and robots.txt are separate. Blocking GPTBot at the edge does not change your robots.txt, and a robots.txt rule does not change what the edge does. Keep the two consistent, as allow AI search, block AI training describes, so a crawler that reads robots.txt gets the same message the edge enforces.

How the excessive-rate rule can catch AI search crawlers

The switch spares OAI-SearchBot, Claude-SearchBot and ChatGPT-User, but Kinsta adds a condition: if any of them generates excessive requests or accesses pages it should not, it may be classified as an excessive-rate AI crawler. At Challenge bots and Challenge everyone, excessive-rate AI crawlers are challenged automatically.

Kinsta does not publish the threshold. It says excessive-rate traffic can come from people using legitimate verified bots or AI tools, from poorly configured AI agents, or from systems spoofing user agents to look legitimate. In practice that means:

  • Kinsta documents the automatic challenge only for Challenge bots and Challenge everyone. Its changelog says Block automations does not add verified bots or aggressive AI crawlers to the blocking.
  • At the two challenge levels, an AI search crawler's access depends on a rate judgment you cannot see in advance.

If AI search visibility matters to you, prefer Block automations as your standing level and use the challenge levels for short incidents.

How to check what GPTBot, OAI-SearchBot and Googlebot receive

A curl request from your laptop with a crawler's user agent does not reproduce what the real crawler gets. Kinsta classifies traffic with its own rules plus Cloudflare's verified bot list and bot scores, which depend on where the request comes from, not only its user agent string. A spoofed GPTBot request is not verified GPTBot. Use it only to spot a blanket user agent rule:

bash
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://www.example.com/

The user agent above is the example OpenAI publishes at the time of writing; the version number changes. Kinsta does not document the status code a blocked request receives, so read whatever comes back alongside the body, and compare it with a normal browser request. To see what the real crawlers get, use MyKinsta's reports:

  1. Open Sites > sitename > Bot protection and read the Request breakdown chart. It covers the last 24 hours by traffic type, and hovering shows Allowed, Challenged (failed verification and blocked) and Blocked counts.
  2. Open Sites > sitename > Analytics > Bot traffic for longer periods, and look at Top traffic, which lists the top paths, user agents, countries and IPs for the last 24 hours. Set its traffic type dropdown to AI crawlers, then to the other types, and read the user agents listed for each.
  3. If OAI-SearchBot, Claude-SearchBot or PerplexityBot appear with challenged or blocked counts, either lower the protection level or add an exception.
  4. Recheck after every change, including after cloning settings from another site, because cloning overwrites the target sites' bot protection settings.

Do not rely on the server log alone. MyKinsta's server logs include an access.log, but Kinsta does not document whether requests blocked or challenged by Bot Protection appear in it. The Bot protection reports are the documented place to see Allowed, Challenged and Blocked counts, so a crawler missing from access.log is not proof that it never came.

How to allow one crawler with exceptions

Kinsta's changelog suggests combining the Block AI crawlers switch with allow exceptions if you want to permit only certain AI crawlers. Add one at Bot protection > Exceptions > Add exception. Each exception matches an IP or CIDR range, a path ("Starts With" only, no wildcards), a user agent or a country, with an action of Allow, Challenge or Block. Kinsta allows up to 50.

Two rules from Kinsta's documentation decide what happens:

  • Allow beats Challenge, and Challenge beats Block, whatever the field. An Allow exception overrides the switch and the protection level for matching requests.
  • Exceptions are per environment. They are not copied between staging and live, and they are not in backups.

Be careful with user agent exceptions. An Allow on PerplexityBot lets any request carrying that string through, including scrapers that copy it, because a user agent is not proof of identity (verifying real AI crawlers covers why). Where the operator publishes IP ranges, an IP exception is the stronger choice. Exceptions do not override Kinsta's vulnerability protection or platform security.

If you run your own Cloudflare zone in front of Kinsta, Kinsta says its bot protection works alongside it only when Cloudflare's WAF and bot features are off; Cloudflare evaluates traffic first, and anything it blocks never reaches Kinsta. Kinsta's bot protection cannot be used with AWS, Microsoft Azure, Sucuri, Fortinet, another CDN or a reverse proxy, because the original traffic source cannot be reliably identified.

Frequently asked questions

Does Kinsta Bot Protection block Googlebot?

Kinsta says Google and Bing crawlers are verified bots that the protection levels do not affect, and the Block AI crawlers switch does not block them. Its documentation does add that any crawler the switch leaves alone can still be challenged at the two challenge levels if it is classed as excessive-rate.

Kinsta says the switch does not block OAI-SearchBot or ChatGPT-User, the crawlers ChatGPT search and user-requested fetches use. It does block GPTBot, which OpenAI uses for training, not for search.

Which protection level is safest for AI search crawlers?

Block malicious traffic or Block automations. Kinsta documents automatic challenges for excessive-rate AI crawlers only at Challenge bots and Challenge everyone, and a request that cannot complete a challenge is blocked.

Can I block GPTBot but allow ClaudeBot on Kinsta?

Kinsta's changelog suggests turning on Block AI crawlers and adding an allow exception for the crawlers you want to permit, and its documentation says exceptions bypass bot protection blocking rules. Prefer an IP-based exception over a user agent one where the operator publishes IP ranges.

Check what crawlers receive after a change

Every change on the Bot protection screen can alter which crawlers reach your pages, and the edge hides its blocks from your server logs. After changing the level or the switch, read the MyKinsta reports above, then run the free single-page check, which needs no account, on an important page to confirm its robots.txt rules and HTML still read the way you expect. If you manage several Kinsta sites for clients, a full-site audit covers up to 500 pages and comes with a PDF of evidence and prioritized fixes.

Filed under CDN and bot protection. Topics: kinsta, bot-protection, wordpress, ai-crawlers, waf