YandexBot and YandexAdditional: Yandex crawlers and AI answers
What the YandexBot user agent does, how to verify it with reverse DNS, and how YandexAdditional rules keep pages out of Search with Yandex AI answers.
By Crawlito · 8 min read
In short
- YandexBot is Yandex's main indexing robot, and its log user agent contains YandexBot/3.0 with a link to yandex.com/bots.
- YandexAdditional and YandexAdditionalBot are robots.txt names Yandex uses to keep page content out of Search with Yandex AI answers; Yandex says they make no indexing requests.
- Yandex says a change for YandexAdditional takes effect when its robots recrawl the page, which typically takes 2 to 14 days.
- A genuine Yandex robot reverse-resolves to a host ending in yandex.ru, yandex.net or yandex.com, and the forward lookup must return the same IP.
- Yandex does not publish its crawler IP list, so DNS checks, not IP allowlists, are the documented way to spot fakes.
On this page
YandexBot is Yandex's main indexing robot. In your logs it shows up as Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots), sometimes followed by a browser version. To keep a page out of Yandex's AI answers without removing it from Yandex search, you don't block YandexBot. You add a Disallow rule for YandexAdditional or YandexAdditionalBot in robots.txt. Yandex describes these as robots that help process robots.txt for Search with Yandex AI and that make no indexing requests of their own.
This post covers the Yandex robots you are most likely to see, what the YandexAdditional rule does and doesn't do, and how to confirm a request really came from Yandex. All the Yandex facts come from its Webmaster help pages, read on 15 November 2026. For the general method of checking crawler IPs across vendors, see how to verify real AI crawlers.
What the YandexBot user agent looks like
Yandex's robot list gives the main indexing robot as:
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)Yandex notes that logs may also show the browser version used for crawling, for example ... AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.268. That version can change, so Yandex recommends leaving it out when you search your logs. Match on the robot name instead:
grep -E "YandexBot/|YandexAdditional|YandexComBot|YandexImages/" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | headYandex runs many other robots. This table lists the ones most relevant to search and AI answers. The last column is Yandex's own answer to whether the robot follows the general rules in robots.txt, meaning a User-agent: * group.
| Robot | User agent contains | Purpose, as Yandex describes it | Follows User-agent: * |
|---|---|---|---|
| YandexBot | YandexBot/3.0 | Main indexing robot | Yes |
| YandexBot (mirror detector) | YandexBot/3.0; MirrorDetector | Finds alternate site addresses | Yes |
| YandexComBot | YandexComBot/3.0; +http://ya.cc/bots | Indexes content for search in languages other than Russian | No |
| YandexImages | YandexImages/3.0 | Indexes images | Yes |
| YandexRenderResourcesBot | YandexRenderResourcesBot/1.0 | Loads resources for rendering pages with JavaScript | No (see below) |
| YandexAdditionalBot | YandexAdditionalBot/1.0 | Helps process robots.txt for Search with Yandex AI answers | No |
| YandexAdditional | YandexAdditional/1.0 | Same purpose as YandexAdditionalBot | No |
Two details stand out. YandexComBot can index content "if there is no explicit robot-specific restriction", so a * group does not stop it. And YandexRenderResourcesBot ignores robots.txt for a resource if the HTML page that uses it is open to YandexBot, but it does not fetch resources for pages that are disallowed.
Yandex also says some of its robots that fetch pages for purposes other than indexing may ignore restrictive User-agent: * rules, so that site owners don't block them by accident. To restrict one of those, you name it in its own group.
What YandexAdditional does for Yandex AI answers
Yandex describes Search with Yandex AI as a search feature that gives a short answer to a query, either automatically or after the user turns it on. According to that page, Yandex AI looks for relevant sources, analyses them in real time with a generative neural network and combines the findings into one answer with references to the sources. Yandex says the answers are generated from pages it has already indexed.
The opt-out is a robots.txt rule. Yandex's example:
User-agent: YandexAdditionalBot
Disallow: /cats/mycat.htmlYandex's robot list describes both YandexAdditionalBot and YandexAdditional as robots that "help process robots.txt" to keep page content out of Search with Yandex AI answers. They apply to pages the main crawler has already indexed and, in Yandex's words, don't make indexing requests. In practice the name works much like a usage token: the rule tells Yandex how it may use content YandexBot has already fetched. Yandex documents the effect only for AI answers and does not say the rule changes how YandexBot crawls or indexes the page.
That puts it in the same family as Google-Extended and Applebot-Extended, with one difference worth knowing. Yandex lists real user-agent strings for YandexAdditional and YandexAdditionalBot, while Google says Google-Extended has no separate HTTP request user agent string. Yandex doesn't say how often, or whether, you should expect those strings in your logs, so don't use their absence as proof that your rule is being ignored.
Three things the Yandex documentation does not say, at the time of writing:
- whether a YandexAdditional rule has any effect on model training. The documented scope is Yandex AI answers only.
- whether Yandex treats the two names differently. They are listed with the same purpose, so naming both is the safe option.
- whether a
User-agent: Yandexgroup covers them. Yandex says aYandexgroup applies to all its robots, but also that some robots ignore it and should be named exactly.
How long it takes
Yandex says the change takes effect once its robots recrawl the page, which typically takes 2 to 14 days. Content may still appear in Yandex AI answers during that window.
How to keep pages out of Yandex AI answers but stay in Yandex search
Give each AI robot its own group and leave YandexBot alone. Because Yandex says these robots don't follow general rules, a Disallow: / under User-agent: * is not a reliable way to reach them.
User-agent: YandexAdditional
Disallow: /
User-agent: YandexAdditionalBot
Disallow: /
User-agent: *
Disallow: /admin/To opt out only part of a site, use paths:
User-agent: YandexAdditional
Disallow: /research/
Disallow: /members/
User-agent: YandexAdditionalBot
Disallow: /research/
Disallow: /members/RFC 9309 lets several User-agent lines share one group, but Yandex's own pages don't document that form: its User-agent page asks for a blank line before every User-agent line, and every example gives each robot a separate group. Repeating the rules per name costs a few lines and avoids the question. Yandex supports Allow, Disallow and Sitemap, as well as its own Clean-param directive. If you also give YandexBot its own group, remember that a crawler uses the group that names it and ignores *, as explained in robots.txt groups. Yandex's User-agent page says the same: if User-agent: Yandex is present, the * group is ignored for Yandex robots.
Write the names exactly as Yandex does. Yandex's robots.txt page says its robot takes the case of robot names and paths into account, even though its User-agent page says case doesn't matter when matching Yandex. Copying the documented spelling avoids the question.
If this is part of a wider policy across vendors, the same logic of keeping search crawlers open while restricting AI use is set out in allow AI search, block AI training.
Don't use robots.txt to remove pages from Yandex search
Blocking YandexBot is a different decision. Yandex warns that pages restricted in robots.txt can still take part in Yandex search. To remove a page from search, Yandex says to use noindex in the HTML or an HTTP header, and not to block the page in robots.txt, or the robot cannot see the instruction. Yandex says its AI answers are generated from indexed pages, so a page removed from the index should drop out as a source too, although Yandex does not state this directly.
Two other Yandex rules catch people out. If robots.txt is larger than 500 KB, or the server does not return 200 OK for it (directly or after a redirect to another robots.txt that returns 200), Yandex considers the site open for indexing. And Yandex stopped taking Crawl-delay into account on 22 February 2018. Use the crawl rate setting in Yandex Webmaster instead.
How to verify YandexBot with reverse DNS
User agents are easy to fake, and Yandex says some robots disguise themselves as Yandex. Its documented check is a reverse lookup followed by a forward lookup:
- Take the IP address from your server log.
- Run a reverse DNS lookup to get the host name.
- Check that the host name ends in
yandex.ru,yandex.netoryandex.com. If it doesn't, the robot is not Yandex's. - Run a forward DNS lookup on that host name. It must return the IP you started with. If it doesn't, the host name is fake.
With dig, the check looks like this (replace the IP with one from your logs):
ip=203.0.113.7
name=$(dig +short -x "$ip" | grep -v '\.arpa\.$' | head -n1 | sed 's/\.$//')
echo "reverse: $name"
case "$name" in
*.yandex.ru|*.yandex.net|*.yandex.com) dig +short "$name" A "$name" AAAA | grep -qxF "$ip" && echo "OK: Yandex" || echo "FAIL: forward lookup mismatch" ;;
*) echo "FAIL: not a Yandex host" ;;
esacThe patterns start with a dot, so a host such as fakeyandex.com fails. Behind a CDN or load balancer, make sure the IP in your log is the client's and not the proxy's before you check it.
Yandex doesn't publish a list of crawler IP addresses. It says its robots use the networks AS13238, AS208722 and AS212066 with addresses that change often, and that reverse DNS filtering holds up better than IP-based rules when its networks change. For a single address, Yandex Webmaster also has an IP address check tool under Tools, which at the time of writing accepts IPv4 addresses and tells you whether one belongs to Yandex indexing bots.
What to do with fake Yandex robots
A request that says YandexBot but fails the DNS check is a scraper borrowing the name. Treat it like any unverified client:
- Block or rate-limit it at the server or CDN, matching on the failed verification, not on the name. A rule that blocks every
YandexBotuser agent also blocks the real one. - Leave robots.txt alone. Fakes don't read it.
- Count only verified requests when you report on crawler activity, as covered in AI crawlers in server logs.
Frequently asked questions
Does blocking YandexAdditional remove my site from Yandex search?
Not according to Yandex's documentation. Yandex says YandexAdditional and YandexAdditionalBot make no indexing requests and describes the rule only as keeping page content out of Search with Yandex AI answers; regular indexing is controlled through YandexBot and noindex.
Does User-agent: * cover YandexAdditional?
Not reliably. Yandex's robot list says these robots don't follow the general rules in robots.txt, so name YandexAdditional and YandexAdditionalBot in their own group.
How long does a YandexAdditional rule take to work?
Yandex says it takes effect when its robots recrawl the page, which typically takes 2 to 14 days.
Does Yandex publish IP ranges for YandexBot?
No. Yandex says its robot IPs change often and are not disclosed, and recommends a reverse DNS lookup to a host ending in yandex.ru, yandex.net or yandex.com, confirmed by a forward lookup.
Does the YandexAdditional rule stop AI training?
Yandex's documentation does not say. The documented scope is Search with Yandex AI answers.
Check your robots.txt as crawlers see it
If you have added YandexAdditional rules alongside rules for other AI crawlers, check that the live file says what you meant and that the crawlers you want can still reach your pages. Run the free single-page check on an important URL as a next step: it needs no account and reads the live site, not the copy in your repository.
Filed under AI crawlers. Topics: ai-crawlers, robots-txt, server-logs, yandex