How to block AhrefsBot, SemrushBot and other SEO crawlers
Block AhrefsBot, SemrushBot, MJ12bot and DotBot with robots.txt tokens from each operator's docs, and see what you lose and how it differs from AI crawlers.
By Crawlito · 7 min read
In short
- AhrefsBot, SemrushBot, MJ12bot and Moz's DotBot all say they obey robots.txt, so a Disallow rule under each token is the documented way to block them.
- Semrush runs about ten separate bots, each with its own token, so a SemrushBot group alone does not stop Site Audit (SiteAuditBot) or the Backlink Audit bot (SemrushBot-BA).
- Semrush and Majestic advise against IP blocking because their crawlers do not use consecutive IP ranges; Ahrefs instead publishes its IP ranges for verification.
- Blocking an SEO tool crawler does not hide links that point to you from other sites, because those links are found by crawling the linking sites.
- SEO tool crawlers feed third-party analytics products, not AI assistants, so blocking them has no direct effect on ChatGPT, Perplexity or Google AI features, with one exception: AhrefsBot data also feeds the Yep search engine.
On this page
- Which SEO crawlers are hitting your site?
- robots.txt tokens for each SEO crawler
- A robots.txt that blocks the main SEO crawlers
- What you lose when you block SEO crawlers
- Why this is a different decision from blocking AI crawlers
- When robots.txt is not enough
- Frequently asked questions
- Check your robots.txt before and after
To block AhrefsBot, SemrushBot and similar SEO tool crawlers, add a robots.txt group for each crawler's token with Disallow: /. All four of the large ones (Ahrefs, Semrush, Majestic and Moz) document that their crawlers obey robots.txt. The catch is that some operators run several bots under different tokens, so you need to list each token you want to stop, and you should know what you give up before you do it.
This guide lists the tokens each operator publishes, gives a copy-ready robots.txt, and explains why this decision is separate from blocking AI crawlers.
Which SEO crawlers are hitting your site?
Before blocking anything, look at what actually reaches you. Search your access logs for the user-agent strings each operator publishes:
grep -Eio "AhrefsBot|AhrefsSiteAudit|SemrushBot[-A-Z]*|SiteAuditBot|SplitSignalBot|MJ12bot|DotBot" access.log \
| sort | uniq -c | sort -rnThe counts show which tools crawl you and how often. If the volume is low and your server is not struggling, you may not need to block anything. Our guide to AI crawlers in server logs covers log locations and counting by day, and the same commands work for SEO bots.
The user agent alone does not prove who sent a request. Ahrefs publishes its crawler IP ranges and says genuine Ahrefs IPs reverse-resolve to hostnames ending in ahrefs.com or ahrefs.net. If a "SemrushBot" ignores your rules, check it is real before blaming Semrush: anyone can copy a user-agent string. Reverse DNS and published IP lists are the usual ways to check.
robots.txt tokens for each SEO crawler
These are the tokens each operator documents on its own bot page, at the time of writing:
| Operator | Token | What it does | Source |
|---|---|---|---|
| Ahrefs | AhrefsBot | Builds the Ahrefs link and content index; also powers the Yep search engine | ahrefs.com/robot |
| Ahrefs | AhrefsSiteAudit | Site Audit crawls run by Ahrefs users | ahrefs.com/robot |
| Semrush | SemrushBot | Backlink Analytics webgraph and many other tools | semrush.com/bot |
| Semrush | SiteAuditBot | Site Audit tool | semrush.com/bot |
| Semrush | SemrushBot-BA | Backlink Audit tool | semrush.com/bot |
| Semrush | SemrushBot-SI | On Page SEO Checker and similar tools | semrush.com/bot |
| Semrush | SemrushBot-SWA | URL checks for SEO Writing Assistant | semrush.com/bot |
| Semrush | SplitSignalBot | SplitSignal SEO A/B tests | semrush.com/bot |
| Semrush | SemrushBot-OCOB | Content Toolkit | semrush.com/bot |
| Semrush | SemrushBot-FT | Plagiarism Checker and similar tools | semrush.com/bot |
| Semrush | SemrushBot-ESI | Enterprise Site Intelligence | semrush.com/bot |
| Semrush | RyteBot | Tools from Ryte, which Semrush lists on its bot page | semrush.com/bot |
| Majestic | MJ12bot | Majestic link index | mj12bot.com |
| Moz | DotBot | Moz link index | Moz DotBot help |
Semrush's list is the one people miss. Blocking SemrushBot stops the main crawler, but the bots behind Site Audit, Backlink Audit, the On Page SEO Checker and the rest have their own tokens and each needs its own User-agent line.
A robots.txt that blocks the main SEO crawlers
Stack the tokens in one group so there is a single set of rules to maintain:
User-agent: AhrefsBot
User-agent: AhrefsSiteAudit
User-agent: SemrushBot
User-agent: SiteAuditBot
User-agent: SemrushBot-BA
User-agent: SemrushBot-SI
User-agent: SemrushBot-SWA
User-agent: SplitSignalBot
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-FT
User-agent: SemrushBot-ESI
User-agent: MJ12bot
User-agent: DotBot
Disallow: /
User-agent: *
Disallow: /admin/Under RFC 9309, a crawler obeys only the group that names it, so these crawlers ignore the * group entirely. That does not matter here because Disallow: / already covers everything. If you only want to keep them out of part of the site, remember that the named group must repeat every rule you want them to follow. robots.txt groups explained walks through that trap.
Some things to know about how each operator reads the file:
- Ahrefs says AhrefsBot may take some time to pick up robots.txt changes. AhrefsSiteAudit obeys robots.txt by default, but Ahrefs lets verified site owners tell it to ignore robots.txt and crawl-delay on their own sites. That only applies to the owner's own audits.
- Semrush says its bot may take up to one hour or 100 requests to notice a robots.txt change. It also says a robots.txt that returns 4xx means no restrictions and a 5xx blocks crawling of the whole site, and that each subdomain needs its own file.
- Majestic says MJ12bot follows a group that names it instead of the
*group, not in addition to it, which matches the RFC. - Moz says DotBot respects the Robots Exclusion Protocol. At the time of writing we could not load Moz's help page directly to confirm its current wording on timing, so test the result in your logs.
Slowing them down instead of blocking
If the problem is load rather than the crawling itself, Crawl-delay is an option for these bots, even though it is not part of RFC 9309 and Google ignores it:
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: MJ12bot
Crawl-delay: 10Ahrefs says AhrefsBot obeys crawl-delay for HTML pages but not for the assets it fetches when rendering. Semrush documents crawl-delay only for SemrushBot, caps it at 10 seconds, and otherwise adjusts its rate to your server's load. Majestic supports values up to 20 seconds.
What you lose when you block SEO crawlers
Blocking these bots is mostly a trade-off with your own tools and other people's reports about you.
- Your own audits stop working. If your team uses Ahrefs Site Audit or Semrush Site Audit, blocking
AhrefsSiteAuditorSiteAuditBotbreaks those crawls. Allow them, or block them only on hosts you do not audit. - Third-party data about your pages gets thinner. Tools can no longer see your outgoing links, page content or on-page changes. Agencies, prospects and partners who use those tools see less about your site.
- Your backlink profile stays visible. Links pointing to you are discovered by crawling the sites that contain them, so blocking a crawler on your own site does not hide who links to you. Competitors can still see them.
- Ahrefs' Yep search engine. Ahrefs says AhrefsBot data powers Yep as well as the Ahrefs platform. If you care about appearing in Yep, blocking AhrefsBot is a cost.
There is no documented effect on Google, Bing or any AI assistant. These crawlers feed commercial SEO products, not the search indexes that AI tools ground their answers in.
Why this is a different decision from blocking AI crawlers
People often lump all bots together, but the purposes differ, and so do the consequences.
| Crawler type | Examples | What blocking changes |
|---|---|---|
| SEO tool crawlers | AhrefsBot, SemrushBot, MJ12bot, DotBot | What SEO tools know about your site |
| AI training crawlers | GPTBot, ClaudeBot, CCBot | Whether future models can be trained on your pages |
| AI search crawlers | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Whether your pages can be shown in that assistant's search answers |
| Search engine crawlers | Googlebot, Bingbot | Whether you appear in Google or Bing, including their AI features |
A common mistake is a "block all the bots" list copied from a forum that puts OAI-SearchBot or PerplexityBot in the same group as AhrefsBot. That turns a decision about SEO tools into a decision about AI search visibility. Keep SEO tool tokens in their own group, and decide on AI crawlers separately using training vs search vs user crawlers and should you block AI crawlers.
When robots.txt is not enough
robots.txt is a request. These operators say they honor it, but it does nothing against impostors using their user-agent strings, or against tools you have never heard of. If you still see unwanted traffic after the change:
- Confirm the requests are genuine. For Ahrefs, check the IP against its published ranges or reverse DNS. For Semrush and Majestic, which say their crawlers do not use consecutive IP blocks, IP verification is harder, so check whether the traffic keeps hitting paths your robots.txt has disallowed for longer than the operator's stated pickup time.
- Block by user agent at the server or CDN for traffic that ignores robots.txt. Block AI crawlers in nginx, Apache and .htaccess has the syntax, and the same rules work for SEO bot tokens.
- Contact the operator. Semrush asks site owners to email bot@semrush.com with the site URL and log entries if its bot ignores the rules for a long time.
Avoid IP blocklists copied from the web. Both Semrush and Majestic say blocking by IP is not effective for their crawlers, and stale lists tend to catch legitimate traffic.
Frequently asked questions
Does blocking AhrefsBot hide my backlinks from competitors?
No. Ahrefs finds links to your site by crawling the pages that link to you. Blocking AhrefsBot on your own domain only stops it reading your pages, including your outgoing links.
Does blocking SemrushBot also block Semrush Site Audit?
No. Site Audit uses SiteAuditBot, a separate token. Semrush lists about ten bots on its bot page, and each one needs its own User-agent line if you want to block it.
Will blocking SEO crawlers affect my Google rankings?
Google does not document any link between SEO tool crawlers and its rankings. These bots collect data for third-party products, and Googlebot is a separate crawler with its own token.
Do SEO crawlers feed ChatGPT or other AI assistants?
None of these operators say their crawl data is used by AI assistants. The one documented secondary use is Ahrefs' Yep search engine, which uses AhrefsBot data.
Check your robots.txt before and after
Blocking SEO crawlers is a one-group change, but a bot list copied from elsewhere can catch AI search crawlers by accident. After editing the file, run each important token through a parser as described in testing robots.txt against each AI crawler. To see how AI crawlers are treated on one of your pages, run the free check.
Filed under AI crawlers. Topics: robots-txt, bot-protection, server-logs, crawl-budget