Hostinger AI crawlers: AI Audit, robots.txt and llms.txt
How Hostinger controls AI crawlers: the CDN AI Audit panel, CDN blocks that override robots.txt, the llms.txt toggles and the AI Builder robots.txt limits.
By Crawlito · 8 min read
In short
- Hostinger's AI Audit sits in hPanel under Performance > CDN > AI Audit and lets CDN users see which AI crawlers visit and block the ones they do not want.
- Hostinger says bots blocked explicitly in CDN settings get a 403 response even when they are verified, and that Under Attack mode and blocked bots can stop crawlers that robots.txt allows.
- On web and cloud hosting, a robots.txt file you create in public_html replaces the virtual robots.txt that WordPress generates.
- Hostinger AI Builder sites get a platform-managed robots.txt that allows all crawlers and cannot be edited, so custom block rules are not possible there.
- On WordPress, llms.txt is switched on in the Hostinger Tools plugin under Hostinger > Tools with the LLM Optimization and Create LLMs.txt file toggles.
On this page
On Hostinger, three layers decide what AI crawlers get from your site: the CDN, which can block a crawler outright with a 403, the robots.txt file, which only asks crawlers to stay out, and an optional llms.txt file. The CDN's AI Audit panel shows which AI crawlers visit and lets you block them, and Hostinger says CDN settings such as blocked bots or Under Attack mode can stop a crawler even when robots.txt allows it. On AI Builder sites you cannot edit robots.txt at all. To keep AI search crawlers in, check the CDN first, then robots.txt, then confirm the result from outside.
Facts below come from Hostinger's announcement and Help Center, read on 15 November 2026. Dashboard menus can change, so confirm the menu path in your own hPanel.
Where Hostinger controls AI crawlers
| Layer | Where | What it does | Enforced? |
|---|---|---|---|
| CDN AI Audit | Websites > Manage site > Performance > CDN > AI Audit | Lists AI crawlers that visit and blocks the ones you choose | Yes, at the CDN |
| CDN Security tab | CDN > Manage > Security | Security level, which sets how aggressively visitors are challenged | Yes |
| Under Attack mode | CDN security settings | Challenges every visitor with a browser verification page | Yes |
| CDN Traffic blocking tab | CDN > Manage > Traffic blocking | Country and IP blocks | Yes |
| Server-level rate limits | Not configurable | 429 for heavy automated traffic from some networks | Yes |
| robots.txt | public_html/robots.txt, WordPress or AI Builder | Asks crawlers to stay out of paths | No, voluntary |
| llms.txt | Hostinger Tools plugin or AI Builder settings | Publishes a curated list of key pages for AI tools | No, informational |
The CDN rows only apply if the CDN is on. Hostinger's CDN article lists it on Premium Web, Business Web, Cloud and Agency hosting, not on Single plans or VPS, and says Hostinger began enabling it automatically for eligible shared hosting sites from 24 August 2026. If you are not sure whether your site sits behind it, look for the x-hcdn-request-id header in a response (see the check section below).
How the AI Audit panel works
Hostinger introduced AI Audit in September 2025 as part of its CDN for web and cloud hosting clients. It lives under Performance > CDN > AI Audit in the website dashboard and does three things: shows which AI crawlers visit the site, tracks their activity and requests over time, and blocks the bots you do not want while keeping the rest.
The announcement's examples of "common AI bots" are Amazonbot, Applebot, GPTBot, Googlebot and Meta-ExternalAgent, and it describes Googlebot as Google's main crawler. It does not publish the full list the panel shows. Two points are worth keeping in mind:
- A block stops future visits only. Hostinger says blocking does not erase what a crawler has already collected.
- Crawler names cover different purposes. Blocking GPTBot is a training opt-out, while blocking OAI-SearchBot or PerplexityBot cuts off AI search, and blocking Googlebot cuts off Google Search. The difference between training, search and user crawlers is the right basis for each choice.
What a blocked crawler receives
Hostinger's AI Audit announcement does not state the response code. Its Help Center article on CDN and search engine crawlers does: bots blocked explicitly in your CDN settings receive a 403 response, even when they are verified. A 403 is a hard stop: the crawler gets an error instead of your page.
The same article says Hostinger CDN verifies crawlers such as Googlebot and Bingbot against the search engines' published IP lists and reverse DNS, so a request that only claims to be Googlebot fails verification. It does not say whether the same verification covers AI crawlers such as OAI-SearchBot or ClaudeBot. Treat that as not documented.
CDN settings that override robots.txt
Hostinger's guide to allowing or blocking AI crawlers says it plainly: robots.txt is a request, and CDN settings such as Under Attack mode and blocked bots can block crawlers even when robots.txt allows them. The settings to review:
- Under Attack mode. Hostinger says it makes the CDN challenge every visitor with a browser verification page, and advises keeping it on only during an active attack. The browser verification page runs a JavaScript check, and clients that cannot complete it, such as most bots, never reach your content. Hostinger says verified search engine crawlers are not shown the page under normal settings, but it does not say the same of AI crawlers. Set the security level back to Medium or Low once an attack ends.
- High security levels. Stricter levels challenge visitors with a higher threat score, and Hostinger says some VPN and data center IP ranges carry higher threat scores, which matters because crawlers fetch from servers, not homes.
- Country and IP blocks. A blocked country also blocks any crawler that fetches from it. Hostinger's example is Googlebot, which mostly crawls from US addresses.
- CDN rate limits. Crawl bursts can return a
429. Hostinger says occasional429responses during bursts are normal, that crawlers retry, and that crawler rate limits cannot be changed from the dashboard, so persistent cases need Hostinger to review them. - Server-level limits. Separately from the CDN, Hostinger documents server-level rate limits that return
429when automated traffic from Meta, AWS and Microsoft network ranges exceeds a limit. They apply to all customers, cannot be turned off per site, and Hostinger does not say whether verified crawlers are exempt.
A challenge or block has the same effect for AI search as a Disallow, only stronger: the crawler reads a verification page or an error instead of your text. Why a WAF challenge page means AI crawlers read nothing explains what the crawler sees.
robots.txt on Hostinger: WordPress, plain hosting and AI Builder
Which file crawlers read depends on how the site is built.
Web and cloud hosting, including WordPress
Hostinger's steps are to open File Manager, go to public_html, open or create robots.txt, add your rules and click Deploy. Its guide also notes that on WordPress, creating the file in public_html replaces the version WordPress generates. That is the same precedence rule covered in editing the WordPress virtual robots.txt: once a physical file exists, rules that an SEO plugin writes to the virtual file are no longer served.
Hostinger's example for opting out of one crawler keeps everything else open:
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /Its guide also shows a Content-Signal: search=yes, ai-train=no line under User-agent: *. That line comes from Cloudflare's Content Signals proposal, not RFC 9309, and states a preference rather than blocking anything. Content signals in robots.txt covers how crawlers treat it.
Hostinger AI Builder
Here you have no choice. Hostinger says that in manual mode robots.txt is generated automatically, allows all crawlers and includes the sitemap, with no setting to edit it. In agentic mode it is created after you publish on a custom domain, it is platform-managed and read-only, and the AI cannot add block rules to it. In both modes custom rules are not supported.
If you need to keep a crawler out of an AI Builder site, robots.txt is not available to you. The CDN block is the documented route, if the site is behind the CDN.
The llms.txt toggles
An llms.txt file is a Markdown file at /llms.txt that lists a site's key pages with short descriptions, following the proposal at llmstxt.org. It does not grant or block access. Hostinger's own llms.txt article calls it a proposed standard that not all AI tools use yet. How to switch it on:
| Site type | Where | Toggles |
|---|---|---|
| WordPress | WordPress admin > Hostinger > Tools (Hostinger Tools plugin) | LLM Optimization on, then Create LLMs.txt file |
| AI Builder, manual mode | Editor > More > General Settings | Web2Agent (Beta) on, then Enable LLMS.txt, then publish |
| AI Builder, agentic mode | Nothing to set | Created automatically on every publish |
Hostinger says the WordPress file is live immediately, the manual mode file may take some time to appear after publishing, and turning the setting off removes the file. Whether any assistant reads it is outside your control. The llms.txt guide covers what the file can and cannot do, so treat it as a small extra after access is sorted, not a fix for a blocked crawler.
How to check what AI crawlers actually get
Dashboard settings rarely show the combined result, so test from outside.
1. Read the robots.txt the world sees:
curl -s https://www.example.com/robots.txtLook for groups naming OAI-SearchBot, Claude-SearchBot or PerplexityBot with Disallow: /, and for a physical file that has replaced the rules you expected.
2. Compare a browser and a crawler user agent, and look for Hostinger's headers:
URL=https://www.example.com/
for UA in \
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
"Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"; do
curl -s -o /dev/null -D - -A "$UA" "$URL" | grep -iE '^(HTTP|x-hcdn-request-id|x-hcdn-cache-status)'
echo
doneHostinger's crawler article uses x-hcdn-request-id to confirm a 429 came through its CDN: when the header is missing, the response came from another layer, such as the website itself. x-hcdn-cache-status shows whether the page came from the CDN cache, and a HIT after a change means you may be testing an old copy. Flush it under Performance > CDN; Hostinger says individual URLs cannot be flushed separately.
Read the result with care. Your request comes from your own IP address, so a test that passes does not prove the real crawler, fetching from a data center, passes too, and a fake-bot rule may block your test but not the verified crawler. The AI Audit panel's request history is the better view of what real crawlers did. How to see your page as an AI crawler covers more tests.
3. If a crawler you allow still gets a 403 or 429, check AI Audit for a block, the CDN security settings for Under Attack mode or a high security level, and the Traffic blocking tab for country or IP rules. For crawl errors, Hostinger asks support requests to include the domain, the exact URL and status code, the approximate time, a screenshot and the x-hcdn-request-id value when available.
Frequently asked questions
Does Hostinger block AI crawlers by default?
Hostinger does not document a default block for AI crawlers. Blocks come from crawlers you block in AI Audit, CDN security settings, country or IP rules, rate limits, or your own robots.txt.
What does a crawler blocked in Hostinger's CDN receive?
Hostinger says bots blocked explicitly in CDN settings get a 403 response, even when they are verified. They get an error instead of your page.
Can I edit robots.txt on Hostinger AI Builder?
No. Hostinger says AI Builder generates a robots.txt that allows all crawlers in manual mode and a read-only, platform-managed file in agentic mode, and custom rules are not supported in either.
Will enabling llms.txt help my site appear in ChatGPT or Perplexity?
Nobody can promise that. llms.txt is a proposed standard that not all AI tools use, and it does nothing if the CDN or robots.txt blocks the crawler, so fix access first.
Check your Hostinger site from outside
After changing AI Audit, the CDN or robots.txt, run the free single-page check, which needs no account, to see the status code and robots.txt rule each main AI crawler gets on one page. For a whole site with evidence and prioritized fixes, an audit covers up to 500 pages.
Filed under Platform guides. Topics: hostinger, ai-crawlers, cdn, robots-txt, llms-txt