PetalBot: what Huawei's crawler is and how to allow or block it
PetalBot is Huawei's Petal Search crawler. What Huawei documents about it, how to verify it, and how to allow, limit or block it in robots.txt.
By Crawlito · 7 min read
In short
- PetalBot is the crawler for Huawei's Petal Search; Huawei says its index also feeds content recommendations in Huawei Assistant and AI Search.
- Huawei documents two user agents, a PC one and an Android mobile one, and both contain the product token PetalBot.
- PetalBot follows robots.txt, so User-agent: PetalBot with Disallow: / blocks it, and blocking it removes your pages from Petal Search.
- Huawei recommends verifying PetalBot with a reverse DNS lookup followed by a forward lookup, because its user agent is often faked.
- Huawei does not document Crawl-delay support; for crawl load it asks site owners to email petalbot@huawei.com.
On this page
- What is PetalBot?
- PetalBot user-agent strings
- How to verify that a request is really PetalBot
- How to block PetalBot in robots.txt
- How to slow PetalBot down without blocking it
- Should you allow or block PetalBot?
- How to check what PetalBot gets from your site
- Frequently asked questions
- Check which crawlers your site lets in
PetalBot is the web crawler for Petal Search, Huawei's search engine. Huawei says it crawls PC and mobile sites to build the Petal Search index, and that the same index powers content recommendations in Huawei Assistant and AI Search. It follows robots.txt, so you can allow it, fence it out of parts of your site, or block it completely with a User-agent: PetalBot group.
If you found PetalBot in your logs and want to decide what to do with it, this guide covers what Huawei actually documents, how to tell the real crawler from impostors, and the robots.txt and firewall options for each choice.
What is PetalBot?
Huawei's PetalBot page on the Petal Search webmaster site describes it as "an automatic program of the Petal search engine". Its job, in Huawei's words, is to access PC and mobile websites and build an index database so users can find your content in Petal Search. The same page says that index is used to present content recommendations in Huawei Assistant and AI Search, which Huawei says are both powered by Petal Search.
That puts PetalBot in the search crawler bucket rather than the training bucket. Huawei does not describe it as collecting data to train models, and it does not publish a separate training token the way Google does with Google-Extended. If you see a third-party list that labels PetalBot an "AI training crawler", check it against Huawei's own wording, which says nothing about training. Our guide to training, search and user-triggered crawlers explains why that distinction matters when you write rules.
Whether Petal Search matters to you depends on whether your customers use Huawei devices and services. Huawei does not publish traffic figures on the PetalBot page, and we do not quote third-party estimates here.
PetalBot user-agent strings
Huawei lists two user agents, one per device type:
| Product name | Full user-agent string |
|---|---|
| PetalBot (PC) | Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot) |
| PetalBot (mobile) | Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) |
Both contain the product token PetalBot, which is the value you use in robots.txt. The verification hostnames Huawei documents use the aspiegel.com domain.
To pull PetalBot requests from an Nginx or Apache access log:
grep -i "petalbot" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | headThat gives you the IP addresses claiming to be PetalBot, ranked by request count. The next step is checking whether they really are.
How to verify that a request is really PetalBot
Huawei warns that "there may be many fake PetalBot crawling" and recommends DNS verification, the same two-step method Google and Bing use:
- Run a reverse DNS lookup on the IP address from your logs.
- Check that the hostname belongs to Huawei's crawler domain.
- Run a forward DNS lookup on that hostname and confirm it resolves back to the same IP address.
host 203.0.113.10
host <hostname-returned-above>Huawei's page is not fully consistent about the hostname. Its text says PetalBot's hostname is in the format petalbot.*.aspiegel.com and that anything else is impersonation, while its worked example shows a hostname of the form petalbot-<ip>.petalsearch.com. In practice, accept a hostname that starts with petalbot and ends in aspiegel.com or petalsearch.com, and require the forward lookup to match. A user agent alone proves nothing, as our guide to verifying real AI crawlers explains.
The Petal Search webmaster site also has an IP check tool ("Verify PetalBot") where you enter an address and it tells you whether it is a verified PetalBot IP. Huawei does not publish a downloadable IP range list on the PetalBot page at the time of writing, so DNS verification or that tool are the documented options.
How to block PetalBot in robots.txt
Huawei states that PetalBot "complies with the Internet robots protocol" and that you can use robots.txt to block all or part of your site. Its own example of a full block:
User-agent: PetalBot
Disallow: /And its example of a partial rule set, which keeps PetalBot out of one folder while allowing an API path:
User-agent: PetalBot
Allow: /w/api/
Disallow: /trap/Two things to keep in mind when you add a PetalBot group:
- It replaces your
*rules for PetalBot. Under RFC 9309 a crawler obeys only the group that names it. If yourUser-agent: *group blocks/cart/or/admin/, repeat those lines in the PetalBot group. Our explainer on robots.txt groups shows how this goes wrong. - Blocking has a cost. Huawei's note is direct: banning PetalBot makes your pages unsearchable in Petal Search and in the services built on it.
Removal is not instant either. Huawei says it can take several months to clear pages already in the Petal Search index after PetalBot stops crawling them, and suggests emailing petalbot@huawei.com if a removal is urgent.
How to slow PetalBot down without blocking it
This is where people most often reach for Crawl-delay. Huawei's PetalBot page does not mention Crawl-delay, so treat it as unsupported unless Huawei documents otherwise. What the page does say is that PetalBot "needs to maintain a certain amount of crawling" and that Huawei adjusts the rate based on server capacity, site quality and how often the site updates. If the load is unreasonable, Huawei asks you to send details to petalbot@huawei.com. The webmaster site also has a "Report Problem" form that asks for the IP address you see, the user agent and how many requests a day you receive.
Practical options, in order of how much they cost you:
- Fence off crawl traps. Faceted filters, internal search results and calendar pages generate endless URLs. Disallowing them for PetalBot (and usually for every crawler) cuts load without losing your real pages. See faceted navigation crawl traps.
- Return 429 or 503 under real overload. Most well-behaved crawlers back off on these codes. Huawei does not document how PetalBot reacts, so watch your logs after you deploy it.
- Rate limit verified PetalBot at the edge. A per-IP limit on requests whose user agent contains
PetalBotlets the crawler continue at a pace your server can handle. Our guide to rate limiting AI crawlers has Nginx and Cloudflare examples. - Contact Huawei through the email address or the report form above.
Should you allow or block PetalBot?
There is no universal answer, but the decision is easier when you separate three cases:
| Situation | Reasonable choice |
|---|---|
| You have customers using Huawei devices or in markets where Petal Search is the default | Allow it, block crawl traps, verify it in logs |
| PetalBot load is causing real problems | Rate limit and contact Huawei before blocking |
| You have no audience on Huawei services and want fewer crawlers | Block it in robots.txt and accept losing Petal Search visibility |
| Requests say PetalBot but fail DNS verification | Block them at the firewall; they are not Huawei |
Blocking impostors at the firewall and blocking the real PetalBot in robots.txt are separate decisions. A firewall rule that matches the user-agent string alone will catch both, and it also hides the real crawler from your robots.txt policy, so match on verified IPs where you can.
How to check what PetalBot gets from your site
Fetch a page with the PetalBot user agent and compare it with a normal browser request:
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
-A "Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)" \
https://www.example.com/A 403, a challenge page or a much smaller response than your browser gets means a firewall or bot rule is treating the user agent differently. Bear in mind that some security products score the IP address as well, so a curl test from your laptop shows the user-agent rule, not everything the real crawler meets.
One last detail worth noticing: Huawei's PetalBot documentation page is itself rendered with JavaScript. A plain HTML fetch of it returns an empty shell with a "doesn't work properly without JavaScript enabled" notice. If your own pages behave the same way, crawlers that read raw HTML will see that shell too; our guide to client-side rendering and empty HTML covers the fix.
Frequently asked questions
Is PetalBot a legitimate crawler or spam?
The real PetalBot is Huawei's search crawler for Petal Search and it follows robots.txt. Fake bots often borrow its user agent, so verify the IP with reverse and forward DNS before deciding it is the real one.
Does PetalBot respect Crawl-delay?
Huawei's PetalBot page does not mention Crawl-delay, so do not rely on it. Huawei says it adjusts crawl rate itself and asks site owners with load problems to email petalbot@huawei.com.
Will blocking PetalBot affect Google or Bing?
No. A User-agent: PetalBot group applies only to PetalBot. Googlebot and Bingbot read their own groups or the * group, so their access is unchanged.
Is PetalBot used to train AI models?
Huawei describes PetalBot as building the Petal Search index, which also feeds recommendations in Huawei Assistant and AI Search. Its documentation does not describe model training, and Huawei does not publish a separate training opt-out token.
How long until blocked pages leave Petal Search?
Huawei says clearing pages already in its index can take several months after PetalBot stops crawling them. For urgent removals it points to petalbot@huawei.com.
Check which crawlers your site lets in
PetalBot is one crawler among many, and the same robots.txt or firewall rule often affects several of them at once. Run the free check on an important page to see which crawlers can fetch it and what they receive, with no account needed.
Filed under AI crawlers. Topics: ai-crawlers, robots-txt, server-logs, bot-protection