Amazonbot vs Amzn-SearchBot vs Amzn-User: what each does
Amazon runs three crawlers. See which one may train Amazon AI models, which follow robots.txt, and how to block training while staying eligible for Alexa search.
By Crawlito · 9 min read
In short
- Amazon says Amazonbot may be used to train Amazon AI models, while Amzn-SearchBot and Amzn-User do not crawl content for generative AI model training.
- Amazon says all three respect the Robots Exclusion Protocol, but Amzn-User may not follow all robots.txt directives because a user can initiate its actions.
- Each Amazon user agent setting is independent, so you can disallow Amazonbot and keep Amzn-SearchBot allowed; Amazon says changes may take about 24 hours to be reflected.
- Amazon says none of the three supports Crawl-delay, and that they respect rel=nofollow plus the page-level robots values noindex, none and noarchive.
- Amazon publishes a separate IP list for each agent but describes no reverse DNS method, so IP matching is the documented way to verify them.
On this page
- What are Amazonbot, Amzn-SearchBot and Amzn-User?
- Which Amazon crawler may be used for AI training?
- How to block Amazonbot but allow Amzn-SearchBot in robots.txt
- Why Amzn-User may not follow every robots.txt directive
- Crawl-delay, nofollow and meta tag support
- How Amazon caches robots.txt and how long changes take
- How to verify Amazon's crawlers with published IP lists
- Frequently asked questions
- Check what your site tells Amazon
Amazon documents three web crawlers. Amazonbot improves Amazon products and services and, Amazon says, may be used to train Amazon AI models. Amzn-SearchBot improves search experiences, and permitting it makes your content eligible to appear in search experiences such as Alexa; it does not crawl for generative AI training. Amzn-User supports user actions, such as answering Alexa queries that need up-to-date information, also without training use, and may not follow every robots.txt directive. To opt out of training and stay eligible for Alexa-style search, disallow Amazonbot and leave Amzn-SearchBot allowed.
Everything below comes from Amazon's crawler page and RFC 9309, as read on 11 November 2026. Amazon can change its page, so treat the details as "at the time of writing". Allowing a crawler makes your pages eligible to be read. It does not mean Amazon will use them in any answer.
What are Amazonbot, Amzn-SearchBot and Amzn-User?
They are three separate user agents with three different jobs. Amazon describes each one in its own words:
| Agent | What Amazon says it is for | Generative AI training | robots.txt |
|---|---|---|---|
Amazonbot | Improves Amazon products and services; may be used to train Amazon AI models | May be used | Respects the Robots Exclusion Protocol |
Amzn-SearchBot | Improves search experiences in Amazon products and services; permitting it makes content eligible to appear in search experiences such as Alexa | Not used | Respects the Robots Exclusion Protocol |
Amzn-User | Supports user actions, such as responding to Alexa queries that need up-to-date information | Not used | Amazon says it "may not follow all robots.txt directives" |
The split follows the same pattern as other operators: one agent that gathers content in bulk, one that feeds search, and one that acts for a person. The training, search and user crawlers guide explains why that matters for your policy, and the AI crawler user agents list puts Amazon next to the other operators.
Amazon's page shows each user agent as a full string. At the time of writing they look like this, with a placeholder where the Chrome version goes:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36Match on the product token (Amazonbot, Amzn-SearchBot, Amzn-User) rather than the whole string, because version numbers change.
Which Amazon crawler may be used for AI training?
Only Amazonbot. Amazon says its use "helps us provide more accurate information to customers and may be used to train Amazon AI models". The wording is "may", and Amazon does not publish which products or models use the data, so you cannot tell from outside whether a given page was used.
For the other two, Amazon is direct. Its page says Amzn-SearchBot "does not crawl content for generative AI model training", and says the same about Amzn-User.
There is a second control besides robots.txt. Amazon lists page-level robots meta tags that its agents respect:
noarchive: do not use the page for model trainingnoindex: do not index the pagenone: do not index the page
<meta name="robots" content="noarchive">Per Amazon, that value asks its agents not to use the page for model training, and it does not appear to ask them to stop reading it. Amazon's page does not show example markup, so the standard form above is our reading of it. The page also describes meta tags only: it does not mention the X-Robots-Tag header, so treat header support as undocumented. The tag also reaches every other crawler that reads it, so check what other operators do with noarchive before using it site-wide.
How to block Amazonbot but allow Amzn-SearchBot in robots.txt
Amazon says each user agent setting is independent of the others, so a rule for one does not carry over to another. Name each agent in its own group:
User-agent: *
Disallow: /admin/
User-agent: Amazonbot
Disallow: /
User-agent: Amzn-SearchBot
Disallow: /admin/Amazonbot is blocked everywhere. Amzn-SearchBot may crawl everything except /admin/.
The Disallow: /admin/ line inside the Amzn-SearchBot group is not decoration. Under RFC 9309, a crawler obeys the group that matches its token and, if none matches, the * group. Once you add an Amzn-SearchBot group, the * rules stop applying to it, so any path you blocked for everyone is open to it unless you repeat the rule. The robots.txt groups guide walks through this trap. Token matching is case-insensitive under the RFC.
Amazon also documents a fallback: if your robots.txt does not mention Amzn-SearchBot but allows other search bots, Amzn-SearchBot follows the rules given to those bots. Amazon does not say which bots count as "other search bots", so do not rely on it. Name Amzn-SearchBot yourself.
Be aware of what you give up. Amazon describes Amazonbot as serving product and service improvement as well as possible training. It documents no training-only token for it, so a Disallow opts out of both. If you want a narrower opt-out, the page-level noarchive tag is the documented alternative. For the same pattern across other vendors, see how to allow AI search but block AI training.
If you use Cloudflare's managed robots.txt, check the file your site actually serves. Cloudflare's documentation says it prepends managed directives only when Bot Preference Sync is on and at least one of your Search, Agent or Training policies is set to block (or, for Training, disallow). Its example output includes Disallow: / for Amazonbot, so you may already be opted out without having written the rule.
Why Amzn-User may not follow every robots.txt directive
Amazon's reason is on its page: "Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives." Amazon does not say which directives it follows and which it skips, so take the sentence at face value.
OpenAI says a similar thing about ChatGPT-User ("robots.txt rules may not apply"), and Perplexity and Meta document comparable user-triggered fetchers. The idea is that a person asked for the page, so the fetch is closer to a visit than a crawl.
What that means for you:
- A
Disallowfor Amzn-User is not a dependable block. You can add one, but Amazon says the agent may not follow it. - If you must stop it, enforce it at the server or CDN. Match the user agent and the published IP list (see the verification section below), and return a
403. - Think before you do. Amazon says Amzn-User supports things like answering Alexa queries that need current information. Blocking it stops those fetches.
Amzn-User does not train models per Amazon, so blocking it is a question of access, not of training opt-out.
Crawl-delay, nofollow and meta tag support
Amazon states what its agents support on one page:
| Signal | Support, per Amazon |
|---|---|
Crawl-delay | "They do not support the crawl-delay directive." |
rel="nofollow" on links | Respected |
noarchive meta robots | Respected: do not use the page for model training |
noindex meta robots | Respected: do not index the page |
none meta robots | Respected: do not index the page |
nosnippet, max-snippet, X-Robots-Tag | Not mentioned on Amazon's page |
The last row means "not documented", not "ignored". Do not assume either way.
Because Crawl-delay is not supported, adding it to your Amazon groups does nothing. Amazon's page does not describe how its crawlers react to rate limits either. If Amazonbot is loading your server, the Crawl-delay guide covers server-side rate limiting, which does not depend on a bot reading a directive.
How Amazon caches robots.txt and how long changes take
Two statements on Amazon's page set your expectations, and they do not match each other:
- The crawlers "will fetch host-level robots.txt files or use a cached copy from the last 30 days."
- Each user agent setting "may take ~24 hours for our systems to reflect changes."
Amazon does not reconcile the two. RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable, so a 30-day window is longer than the RFC's guidance. Amazon gives no way to tell which applies to your site, so if you add a Disallow today, allow at least a day for it to take effect and do not be surprised by requests for longer. Your logs will show whether requests stop.
Two more details from the same section:
- Each host has its own file. Amazon says it honors the robots rules exposed under each host, so
shop.example.comneeds its own/robots.txtand the one onwww.example.comdoes not cover it. - A file it cannot fetch is treated as absent. Amazon says that when a file can't be fetched, its agents behave as if it does not exist. RFC 9309 says the opposite for server errors: a
5xxshould be read as "disallow everything". Amazon does not say whether its sentence covers5xxresponses specifically. The safe reading is that a robots.txt that fails during a deploy may leave you open to Amazonbot even though you blocked it.
Check the status code of your file:
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/robots.txtHow to verify Amazon's crawlers with published IP lists
A user agent is only a claim, as the guide to verifying AI crawlers explains. Amazon publishes one IP list per agent:
| Agent | List |
|---|---|
| Amazonbot | developer.amazon.com/amazonbot/ip-addresses/ |
| Amzn-SearchBot | developer.amazon.com/amazonbot/searchbot-ip-addresses/ |
| Amzn-User | developer.amazon.com/amazonbot/live-ip-addresses/ |
Three things were not what we expected when we fetched them on 11 November 2026:
- They are web pages, not standalone
.jsonfiles. The server returnedtext/htmlwith the JSON shown inside the page, so a plain JSON parser fails on the response. - The field names differ. The Amzn-SearchBot list uses
ip_prefixwith/32suffixes. The Amazonbot and Amzn-User lists useipv4Prefixwith bare addresses and no mask. - The
creationTimevalues differ. The Amazonbot and Amzn-SearchBot lists showed 8 September 2026, and the Amzn-User list showed 4 November 2025. Amazon does not say how often the lists change, so fetch them on a schedule instead of pasting addresses into a config.
The entries we saw were single IPv4 addresses, roughly 800 to 1,300 per list. Amazon's page describes no reverse DNS check, so IP matching is the only documented method. This command checks one address against the Amzn-SearchBot list and handles both field names:
curl -s --retry 3 https://developer.amazon.com/amazonbot/searchbot-ip-addresses/ | python3 -c '
import ipaddress, re, sys
ip = ipaddress.ip_address(sys.argv[1])
found = re.findall(r"\"(?:ip_prefix|ipv4Prefix|ipv6Prefix)\"\s*:\s*\"([^\"]+)\"", sys.stdin.read())
print(any(ip in ipaddress.ip_network(p, strict=False) for p in found))
' 203.0.113.7Swap in one of the other two URLs for Amazonbot or Amzn-User. True means the address is on that list. False for a request that claims the agent means it is fake or the list has changed since you fetched it.
To see which agents are visiting at all, count them in your access log:
grep -oE "Amazonbot|Amzn-SearchBot|Amzn-User" access.log | sort | uniq -cThen check the source IPs of the hits against the matching list before you trust the numbers.
Frequently asked questions
Does blocking Amazonbot remove me from Alexa answers?
Amazon says permitting Amzn-SearchBot makes your content eligible to appear in search experiences such as Alexa, and that each agent's settings are independent. Disallowing Amazonbot alone therefore does not change Amzn-SearchBot's rules. Eligibility is not a promise that Amazon will use your page.
Does Amazon say Amazonbot definitely trains its models?
No. The wording is that Amazonbot "may be used to train Amazon AI models". Amazon does not say which pages, products or models are involved.
Can I stop Amzn-User with robots.txt?
Not reliably. Amazon says it may not follow all robots.txt directives because users can initiate its actions. Use a server or CDN rule that matches the user agent and Amazon's published IP list if you need to stop it.
How do I tell the three Amazon agents apart in my logs?
Search for the product tokens Amazonbot, Amzn-SearchBot and Amzn-User in the user agent. Then verify the source IP against the matching list, since anyone can send those names.
Check what your site tells Amazon
Copy the robots.txt example above, then confirm what your site actually serves, including any rules your CDN adds. The free check reads one page without an account and shows the robots rules, headers and raw HTML a crawler receives.
Filed under AI crawlers. Topics: amazonbot, ai-crawlers, robots-txt, server-logs