Skip to content

SiteGround AI crawlers: server-level blocks and how to test them

SiteGround blocks AI training crawlers at server level and allows live-chat crawlers. How to test which bots get a 403, read the cache header and ask for changes.

By Crawlito · 8 min read

In short

  • SiteGround says it blocks AI crawlers designed to scrape content for model training, and allows by default the crawlers used when real people interact with AI platforms such as ChatGPT, Claude and Gemini.
  • A host-level block works before robots.txt is consulted, so a robots.txt that allows a crawler does not unblock it; changes go through a SiteGround support request under the AI crawler setup option.
  • SiteGround's own .htaccess example blocks ChatGPT-User, Claude-User and Perplexity-User, the user-triggered agents its server policy lets through, so copy it only if that is what you want.
  • Dynamic Cache is on by default and cached pages skip .htaccess rules; a response with x-proxy-cache: HIT came from the cache, so flush it before you trust a test.
  • A curl test with a crawler user agent proves how your rules treat that string, not what the real crawler receives from its own IP addresses.
On this page

SiteGround blocks some AI crawlers on its servers before your site is involved. Its published position is that crawlers built to scrape content for AI model training are blocked, while crawlers used when a real person asks an AI assistant to read a page are allowed by default. Because the block sits at the host, a robots.txt that allows a crawler cannot lift it: you have to ask SiteGround support. This post shows how to test which crawlers get through, how to tell a host block from your own rules, and how SiteGround's cache changes what your tests show.

The SiteGround facts below come from its AI bot crawling announcement, dated 28 August 2025, and its knowledge base article How to block AI crawlers?, last updated 2 November 2026. Both were read on 15 November 2026. For the WordPress settings that sit on top of the host (search visibility, SEO plugins, security plugins), see the WordPress AI crawler guide.

What SiteGround blocks and allows by default

SiteGround describes its policy by crawler purpose, not by a public list of names:

Crawler purposeSiteGround defaultExamples by purpose (not SiteGround's list)
Training: scraping content for AI modelsBlocked at server levelGPTBot, ClaudeBot, CCBot
Live chat sessions: a person asks an assistant to visit a pageAllowedChatGPT-User, Claude-User, Perplexity-User
Aggressive AI crawlersBlockedNot named
All bot trafficMonitored and rate limitedNot named

The announcement says the crawlers used "when real users interact with AI platforms" are allowed by default, and names ChatGPT, Claude and Gemini as examples of those platforms. It refers readers to a knowledge base list of which crawlers are allowed by default and which can be enabled on request. At the time of writing, the knowledge base links given for that list (/kb/ai-crawlers-siteground-servers and /kb/allowed-ai-crawlers) show the "How to block AI crawlers?" article instead, which does not say which crawlers the server blocks.

So two things are not documented publicly right now: the exact list of blocked crawler names, and where search-index crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fall. They are neither training crawlers nor live chat fetchers, as training, search and user crawlers explains. Do not assume either way. Test them.

Why robots.txt does not change a SiteGround block

robots.txt is a file a polite crawler reads and then decides to follow. A server-level block happens before that decision: the request is refused regardless of what your robots.txt says. That is why a SiteGround site can have a robots.txt that allows GPTBot and still return nothing useful to it.

Two consequences follow:

  1. A Disallow never produces a 403. If a crawler gets a 403 Forbidden or another error status for a page, robots.txt did not cause it. Something on the server answered: the host, an .htaccess rule, a security plugin or a firewall.
  2. Allowing a crawler takes two steps. Allow it in robots.txt, then make sure nothing at the server refuses it. On SiteGround the second step can mean a support request. The WordPress robots.txt guide from SiteGround covers the first step, including the fact that a physical robots.txt in public_html overrides the virtual one WordPress generates.

How to test which AI crawlers get a 403

Before testing, flush Dynamic Cache, because a cached page can answer the request and skip the server rules (more on that below). SiteGround's Dynamic Cache tutorial gives the path: Site Tools > Speed > Caching > Dynamic Cache, then Flush Cache.

Then send the same request with a set of crawler user agents and read the status codes:

bash
URL=https://www.example.com/a-real-post/

for ua in \
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 Safari/605.1.15" \
  "GPTBot/1.4" "OAI-SearchBot/1.4" "ChatGPT-User" \
  "ClaudeBot/1.0" "Claude-SearchBot" "Claude-User" \
  "PerplexityBot/1.0" "Perplexity-User/1.0" "CCBot/2.0"; do
  curl -s -o /dev/null -D - -A "$ua" "$URL" \
    | awk -v ua="${ua:0:60}" 'NR==1{code=$2} tolower($1)=="x-proxy-cache:"{c=$2} END{printf "%-60s %s %s\n", ua, code, c}'
done

Each line shows the user agent, the status code and, when present, the x-proxy-cache value. Read it like this:

ResultWhat it most likely means
Browser 200, crawler 403A user agent rule refused the crawler: the host, your .htaccess or a plugin
Browser 200, crawler 200 with x-proxy-cache: HITThe page came from Dynamic Cache, so no rule was checked. Flush and retest
Browser 200, crawler 200 without a cache hitNothing refused that user agent string from your IP
Browser and crawler both 403A broader block, such as an IP or country rule, not a crawler rule

Keep the limit of this test in mind. Your request comes from your own network. A host can treat a request that claims to be GPTBot from a home IP differently from the real GPTBot coming from OpenAI's published ranges, and SiteGround does not document how its block identifies crawlers. A 200 here proves only that the user agent string alone is not refused. See your page as an AI crawler covers the method in more depth, and verifying real AI crawlers shows how to check the source IP of requests in your logs.

How to tell the host block from your own rules

A 403 for a crawler on SiteGround can come from three places. Work through them in order:

  1. Your .htaccess file. Open Site Tools > Site > File Manager, go to public_html and read .htaccess. Look for RewriteCond %{HTTP_USER_AGENT} lines that name the crawler, and for rules a plugin wrote between marker comments. If you find one, that is your block, not SiteGround's.
  2. A WordPress security plugin. Check its blocked bots list and its log of blocked requests. A plugin block happens inside WordPress, so a static file such as an image or a core JavaScript file usually still returns 200 to the same user agent. Do not use /robots.txt for this comparison unless a physical file exists: WordPress generates the virtual one itself, so plugins can affect it. Security plugins that block crawlers covers the common ones.
  3. The SiteGround server. If .htaccess and plugins contain nothing that names the crawler and it still gets a 403, the host is the likely source. Compare a few paths:
bash
UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
for path in / /robots.txt /wp-includes/js/jquery/jquery.min.js /this-page-does-not-exist; do
  printf '%-30s %s\n' "$path" "$(curl -s -o /dev/null -w '%{http_code}' -A "$UA" "https://www.example.com$path")"
done

A refusal on every path, including the core JavaScript file and the URL that should return 404, points to a rule that runs before WordPress. If NGINX Direct Delivery is on, static files skip .htaccess (see below), so a 403 on the JavaScript file too points away from your own .htaccess rules. Your server access logs add the real-world view: count the status codes returned to each crawler token over a few days. If the real crawler's requests show 403 while robots.txt allows it and your own rules are clean, ask SiteGround.

How to ask SiteGround to enable or block a crawler

SiteGround's team, replying to comments on the announcement, points customers to a support request: in the Client Area, open the Help Center, choose Contact Us, then Other, and pick the AI crawler setup option. The replies spell the option slightly differently, so look for the one that mentions AI crawlers. The team says it can help with blocking as well as with the crawlers that can be enabled on request.

When you write the request, include:

  • the domain and the exact URL you tested
  • the crawler names you want enabled or blocked, spelled as tokens: OAI-SearchBot, not "ChatGPT"
  • the status code and response headers you saw for each one
  • the date and time of the test, so support can match it in the server logs

Ask support to confirm which crawlers are refused at server level on your account. That answers the question the public documentation leaves open.

Blocking a crawler yourself with .htaccess

To block a crawler that SiteGround lets through, its knowledge base documents an .htaccess rule placed at the very top of the file in public_html, after you download a backup copy. This is SiteGround's multi-bot example:

apache
# Block several AI crawlers
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(ChatGPT-User|Claude-User|Perplexity-User).*$ [NC]
RewriteRule .* - [F,L]

Notice what the example blocks: the user-triggered agents, which are exactly the crawlers SiteGround's server policy allows for live chat sessions. That is a valid choice if you do not want assistants fetching pages on a person's behalf, but it means those agents get a 403 when someone asks ChatGPT, Claude or Perplexity to read your page. Swap in the tokens you actually want to refuse. SiteGround's own crawler table warns that blocking search or user-triggered bots can keep a site out of that assistant's answers, and that Googlebot does not belong in these rules.

The example also blocks /robots.txt for the matched agents. The nginx and .htaccess guide shows how to exclude robots.txt from the rule and why that matters.

Test with SiteGround's documented command, then repeat without the user agent to confirm normal visitors still get a 200 or a redirect:

bash
curl -I -A "ChatGPT-User" https://www.example.com/
curl -I https://www.example.com/

The Dynamic Cache and NGINX Direct Delivery caveat

SiteGround documents two cases where .htaccess rules are skipped:

  • Dynamic Cache, on by default. Once a page is in the cache, every visitor gets it, blocked crawlers included. A response with x-proxy-cache: HIT came from the cache, and SiteGround's advice is to flush the cache and test again. It follows that the rules apply to a page again only until it is cached once more.
  • NGINX Direct Delivery. When it is on, NGINX serves static files such as images, CSS and JavaScript directly, and .htaccess rules do not apply to them.

In practice this means an .htaccess block on SiteGround is partial: a blocked crawler can still receive any page that is cached at the moment it asks. It also distorts tests in both directions. A 200 with a cache hit says nothing about your rules, and a 403 right after a flush may turn into a 200 later, once the page is back in the cache. SiteGround's article notes that robots.txt has no such gap for crawlers that follow it, because they read it before requesting pages, so keep a robots.txt Disallow alongside any .htaccess block.

SiteGround does not document whether its own server-level AI crawler block runs before or after Dynamic Cache, so do not assume a cached page reaches a crawler the host blocks.

Frequently asked questions

Does SiteGround block GPTBot?

SiteGround says it blocks crawlers designed to scrape content for AI training, and GPTBot is OpenAI's training crawler. Its public pages do not list the blocked names at the time of writing, so test with curl and ask support to confirm for your account.

ChatGPT search relies on OAI-SearchBot, not GPTBot, and SiteGround's documentation does not say how it treats search-index crawlers. Test OAI-SearchBot specifically and ask support to enable it if it is refused.

I allowed AI crawlers in robots.txt. Why do they still get a 403?

robots.txt cannot cause or lift a 403. The refusal comes from the server: SiteGround's block, an .htaccess rule or a security plugin. Check those three in that order.

Why does my .htaccess block not work on some pages?

Dynamic Cache serves cached pages without checking .htaccess, and NGINX Direct Delivery serves static files the same way. Look for x-proxy-cache: HIT in the response, flush the cache in Site Tools and test again.

Check what crawlers receive from your site

After any change on SiteGround, run the free check on an important page. It needs no account and shows the robots.txt rule for each named AI crawler and how the page responds. For a whole site, an audit reviews up to 500 pages and returns the evidence and prioritized fixes you can attach to a support request.

Filed under Platform guides. Topics: wordpress, ai-crawlers, bot-protection, apache, siteground