Skip to content

Hetzner blocked user agents: why GPTBot and ClaudeBot get 403

Hetzner blocks GPTBot, ClaudeBot, SemrushBot and five other user agents with a 403 on web hosting and managed servers. How to confirm it and ask for a change.

By Crawlito · 8 min read

In short

  • Hetzner blocks eight user agents by default on all its web hosting and managed server products: Amazonbot, Bytespider, ClaudeBot, GPTBot, MJ12bot, meta-externalagent, Scrapy and SemrushBot.
  • The block is enforced at the server level with HTTP 403 and works independently of robots.txt, so an Allow rule for GPTBot does not change the result.
  • OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User are not on Hetzner's list, so ChatGPT search and Claude search crawling are not stopped by this block.
  • Customers cannot edit the list themselves: Hetzner asks for an authorized support request through konsoleH naming the user agents to allow and the domain.
On this page

If GPTBot, ClaudeBot or SemrushBot get 403 Forbidden from a site on Hetzner web hosting or a Hetzner managed server, the most likely cause is Hetzner's own default block list, not your robots.txt or your CMS. Hetzner documents eight user agents that it refuses at the server level on all of these products, and you cannot change the list yourself. To lift the block for a domain, you send an authorized support request through konsoleH. The AI search and user-triggered agents from OpenAI and Anthropic are not on the list.

This post covers what Hetzner blocks, how to confirm that the 403 comes from that list, what still gets through, and how to decide whether to ask for a change. If you want to write your own user agent rules instead, see blocking AI crawlers in nginx or .htaccess.

Which user agents does Hetzner block?

Hetzner's page on blocked user agents says the block is enabled by default on all its web hosting and managed server products, because these bots have repeatedly caused excessive load on customers' sites. At the time of writing (the page was last changed on 18 May 2026), the list is:

User agentOperatorTypical use, as Hetzner describes it
AmazonbotAmazonImproving Amazon products and services; Amazon says content may also train AI models
BytespiderByteDanceCrawling, including for AI-related applications
ClaudeBotAnthropicCollecting content that may contribute to AI model training
GPTBotOpenAICrawling content that may be used to train OpenAI models
MJ12botMajesticBuilding a link database for SEO analysis
meta-externalagentMetaCrawling for AI-related applications
ScrapyOpen-source projectA framework for custom crawlers, so the name says nothing about who runs it
SemrushBotSemrushSEO, website and marketing analysis

Two of these are SEO tools rather than AI crawlers. If your agency's backlink or audit data looks thin for a Hetzner-hosted client, this list is a likely reason. The MJ12bot guide and blocking AhrefsBot and SemrushBot explain what those two bots do.

Why robots.txt does not fix a Hetzner 403

Hetzner states that the block is enforced at the server level with an HTTP 403 response and is independent of robots.txt. Its reasoning is that robots.txt tells crawlers what they should not crawl but is not a technical access control, so a bot that ignores it could still cause load.

The practical consequence runs the other way too: an Allow rule does not open the door. This robots.txt changes nothing for GPTBot on a Hetzner product with the default block in place:

User-agent: GPTBot
Allow: /

GPTBot still receives a 403 for your pages. Editing robots.txt, clearing a cache plugin or switching off a WordPress security plugin should not help either, because Hetzner applies the block at the server level rather than in your site's own files. If you have checked robots.txt for the usual mistakes and found nothing, the hosting layer is the next place to look.

One detail is not documented: whether the block also applies to /robots.txt itself. That matters, because a crawler that cannot read your robots.txt never sees your rules. The curl test below shows you which is the case on your site.

How to confirm the 403 comes from Hetzner's list

You want to see a pattern: the eight listed names get a 403, while similar names that are not on the list get a normal response. Run this from any machine:

bash
site=https://example.com

for ua in "GPTBot/1.4" "ClaudeBot/1.0" "SemrushBot/7~bl" "MJ12bot/v1.4.8" \
          "OAI-SearchBot/1.4" "ChatGPT-User/1.0" "Claude-SearchBot" "Claude-User" \
          "PerplexityBot/1.0" "Mozilla/5.0 (Macintosh) Safari/605.1.15"; do
  printf '%-42s page %s  robots.txt %s\n' "$ua" \
    "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" "$site/")" \
    "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" "$site/robots.txt")"
done

How to read the result:

  • 403 for the first four, your normal status (200, or your usual redirect) for the rest: this matches Hetzner's list. The block is working as documented.
  • 403 for the search agents too: something else is refusing them, such as your own .htaccess, a plugin or a CDN in front of the server. Hetzner's list does not include them.
  • 403 for the browser line: the problem is not bot-specific at all. Check the site's general configuration first.
  • The robots.txt column: tells you whether blocked crawlers can still read your rules.

The version numbers in the loop are only examples; OpenAI notes that its version numbers may change. These short strings are enough if the server matches on the name. Hetzner does not document whether it matches the exact token or any string that contains it, so if a related agent matters to you, test it directly. Semrush, for example, documents separate agents such as SiteAuditBot and SemrushBot-SI on its bot page, and the curl loop is the only way to see how your server treats them.

Remember that curl sends the name from your own IP. It proves how the server reacts to the header, not what the real crawler experiences. Seeing your page as an AI crawler covers that limit. For real traffic, look in your access log for lines whose user agent contains GPTBot or ClaudeBot and check the status column: a steady run of 403 entries confirms the block is hitting the real bots.

What still gets through: search and user-triggered agents

The block targets one agent per operator, and most operators run several. Hetzner's page says that OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User are not blocked. Checking the operators' own documentation shows what that means.

OperatorBlocked by HetznerNot on the listWhat the unblocked agent does
OpenAIGPTBotOAI-SearchBot, ChatGPT-UserSearch indexing for ChatGPT, and fetches a user asks for
AnthropicClaudeBotClaude-SearchBot, Claude-UserSearch indexing for Claude, and fetches for a user's question
AmazonAmazonbotAmzn-SearchBot, Amzn-UserSearch experiences such as Alexa, and live fetches for Alexa queries
Metameta-externalagentmeta-webindexer, meta-externalfetcherMeta AI search result quality, and links fetched at a user's request

The sources for each row:

  • OpenAI. Its crawler documentation says disallowing GPTBot indicates a site's content should not be used for training, that each crawler setting is independent of the others, and that ChatGPT-User is not used to determine whether content may appear in search. Sites opted out of OAI-SearchBot are not shown in ChatGPT search answers.
  • Anthropic. Its help article describes ClaudeBot as collecting content that could help train its models, Claude-SearchBot as crawling to improve search results, and Claude-User as fetching pages when a user asks a question.
  • Amazon. Its Amazonbot page says Amzn-SearchBot and Amzn-User do not crawl for generative AI training.
  • Meta. Its web crawler page describes meta-externalagent as crawling for uses such as training foundation AI models, meta-webindexer as improving Meta AI search results, and meta-externalfetcher as fetching individual links at a user's request. Meta adds that meta-externalfetcher may bypass robots.txt rules.

PerplexityBot is not on Hetzner's list either. In short, the default block is mostly crawlers that collect content for AI models, plus two SEO crawlers and the Scrapy framework. The difference between training, search and user crawlers explains why that split matters. If the curl test shows a 403 for any search or user agent, look for a second layer of blocking on your own setup.

Should you ask Hetzner to allow GPTBot or ClaudeBot?

That depends on what you want from the training crawlers. The block does not stop ChatGPT search or Claude search from crawling your site, so leaving it in place is a reasonable default for most sites. Consider asking for a change when:

  • You want your content available for model training. Some publishers and documentation sites choose to allow GPTBot and ClaudeBot. With the default block they cannot read your pages, whatever robots.txt says.
  • You rely on SEO tools for your own site. If you use Semrush or Majestic data for a domain on Hetzner, the block on SemrushBot or MJ12bot limits what those tools can collect.
  • You want robots.txt to be the single source of truth. With the block lifted, your robots.txt rules decide again, and you can still disallow the bots there.

Keep in mind why Hetzner added the block: its page says these bots have caused excessive load. If you lift it and then see load problems, rate limiting them with 429 responses is a gentler option than a full block, where your setup lets you configure it. Hetzner's page does not describe rate limiting as an option, so ask support what your product allows.

How to request a change through konsoleH

Hetzner's page is direct about this: you cannot customize the block list yourself, and the managed server support team can adjust it for you. Hetzner asks for an authorized support request through konsoleH. In practice that means:

  1. Log in to konsoleH with an account that is authorized to submit support requests for the product.
  2. Open a support request and say that you want user agents from the default block list allowed.
  3. Name the exact user agents, such as GPTBot and ClaudeBot.
  4. Name the domain or domains the change should apply to. Hetzner asks for both the agents and the domain.

After support confirms the change, run the curl loop again. The previously blocked names should now get the same status as the browser line. Then decide what you want those bots to do and say it in robots.txt, because that file now has the final word for crawlers that respect it. A ready-made file that keeps AI search open while opting out of training is in allow AI search, block AI training.

Hetzner does not document how long a change takes or whether the list will grow, so recheck the page and your curl results from time to time.

Frequently asked questions

Not by itself. Hetzner blocks GPTBot, which OpenAI ties to model training, and leaves OAI-SearchBot and ChatGPT-User unblocked. OpenAI says sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, and Hetzner does not block that agent.

Can I allow GPTBot with robots.txt or .htaccess on Hetzner?

Not with robots.txt: Hetzner says its 403 is enforced at the server level and independent of robots.txt. Hetzner does not document .htaccess as a way around it; the only documented route is an authorized support request through konsoleH.

Does the block apply to Hetzner Cloud servers?

Hetzner's page says the block is enabled on its web hosting and managed server products. It does not mention unmanaged cloud or dedicated servers, where you run the web server yourself.

Is PerplexityBot blocked on Hetzner?

PerplexityBot is not on Hetzner's documented list. If it gets a 403, check your own rules, plugins and any CDN in front of the site.

Check what your Hetzner site returns to AI crawlers

After you confirm or change the block, run the free check on one page, with no account needed, to see what it returns. If you want the same question answered across a whole site, an audit covers up to 500 pages and comes with a PDF of evidence and prioritized fixes.

Filed under CDN and bot protection. Topics: bot-protection, gptbot, claudebot, http-status, troubleshooting