Skip to content

MJ12bot: verify Majestic's crawler and read its robots.txt rules

MJ12bot crawls from many IPs, so Majestic offers a private ident header to verify it. How it reads robots.txt, 403s, redirects and nofollow links.

By Crawlito · 9 min read

In short

  • MJ12bot is Majestic's distributed link crawler; Majestic does not publish IP ranges, so its documented way to verify requests is a pre-arranged ident string sent in a CRAWLER-IDENT header or the user agent.
  • Majestic says that if MJ12bot cannot retrieve robots.txt it assumes it may crawl the site, while certain fetch failures such as 403 Forbidden are treated as a blanket disallow.
  • MJ12bot follows redirects for robots.txt only within the same domain, so a robots.txt that redirects to another domain can look like the bot ignoring your rules.
  • Majestic says MJ12bot crawls links marked rel=nofollow, and that robots.txt Disallow is the way to keep it off a page.
  • Majestic lists three common causes of false violation reports: off-site robots.txt redirects, shared access logs across domains, and a robots.txt tested on a server that differs from the live site.
On this page

MJ12bot is the crawler behind Majestic, a UK backlink index. To confirm a request really comes from it, ask Majestic for a pre-arranged ident string: Majestic says it will send that string only to the domain you name, either in a CRAWLER-IDENT request header or inside the user agent. There is no published IP list to check against, because MJ12bot runs on a distributed network. For robots.txt, the details that trip people up are that an unreachable file is read as "crawl allowed", a 403 is read as "disallow everything", and redirects are followed only on the same domain.

Everything below comes from Majestic's bot page at mj12bot.com, read on 2026-11-15. If you only want the robots.txt group that blocks MJ12bot alongside AhrefsBot and SemrushBot, the SEO crawler blocking guide has it. This post covers what that guide does not: verification, failure handling and the reasons a block can seem not to work.

What MJ12bot is and why it comes from so many IPs

Majestic is a trading name of Majestic-12 Ltd, a company registered in England. Its about page describes the product as a link intelligence database built by surveying and mapping the web. MJ12bot is the crawler that feeds it.

Majestic's bot page says:

  • MJ12bot "does not currently cache web content or personal data". The crawled data is "currently only a web graph of links", added to Majestic's Site Explorer backlinks index.
  • The crawler is "a community based distributed crawler", run on machines contributed by people with broadband connections. That is why Majestic says it cannot restrict the bot to a limited number of IP addresses, and why it asks site owners not to block it by IP in .htaccess.
  • MJ12bot supports Crawl-Delay up to 20 seconds, and requests can still arrive from several MJ12bot instances at once.

The page does not publish a full user-agent string. It lists version numbers, with v1.4.8 marked as current since April 2017, so match on the token MJ12bot in your logs rather than on one exact string.

MJ12bot is an SEO tool crawler, not an AI crawler. Majestic does not say its link graph is used by any AI assistant, so blocking or slowing it is a separate decision from your AI crawler rules. The token, Crawl-Delay cap and the case against IP blocking are covered in the SEO crawler guide, and how Crawl-Delay works across crawlers is in Crawl-delay for AI bots.

How to verify MJ12bot with the CRAWLER-IDENT header

For most named crawlers you verify a request with reverse DNS or a published IP list. Majestic offers neither. Its alternative, in its own words, is that it "can send a pre-arranged ident string with all requests to your site".

How it works, according to the bot page:

  1. You email bot@majestic12.co.uk with details of your site and the ident you want sent. If you prefer, Majestic can generate a random ident for you.
  2. Majestic sends that string with requests to your site, either in an HTTP or HTTPS header field named CRAWLER-IDENT or as part of the user-agent string.
  3. Majestic says it will not share the string with anyone else or send it to any domain or subdomain other than the one you asked for. A request that carries it can therefore be treated as coming from Majestic's network.

The page does not say how long setup takes, whether one ident can cover several subdomains, or how to change an ident later. Ask when you write in, and name every host you want covered.

Checking the header in nginx

Once you have an ident, you can tag requests that claim to be MJ12bot but do not carry it. nginx exposes a request header as a variable in lowercase with hyphens turned into underscores, so CRAWLER-IDENT becomes $http_crawler_ident. Replace your-ident-string with the value you agreed.

nginx
# http block
map "$http_user_agent|$http_crawler_ident" $mj12_unverified {
    default                          0;
    "~*MJ12bot.*\|your-ident-string$"  0;
    "~*MJ12bot"                        1;
}

log_format mj12 '$remote_addr [$time_local] "$request" $status '
                'host=$host verified=$mj12_unverified "$http_user_agent"';

# server block
access_log /var/log/nginx/mj12bot.log mj12 if=$mj12_unverified;

nginx checks the regular expressions in order and uses the first match, so a request with MJ12bot in the user agent and the agreed ident gets 0 and any other MJ12bot request gets 1. If your ident contains characters with a meaning in regular expressions, such as . or +, escape them. Keep your existing access_log line in the same server block: nginx only inherits access_log from the http level when a block defines none of its own.

This writes only unverified "MJ12bot" requests to a separate log, so you can see how much of that traffic is real before acting on it. If you chose to have the ident placed in the user agent instead of the header, match the user agent alone. Logging first is safer than blocking: if the header ever stops arriving, a block would lock out the genuine crawler. Our guide to identifying an unknown bot user agent covers what to do with the requests that fail.

Without an ident, the next best evidence is behaviour. A genuine MJ12bot reads robots.txt and stops requesting disallowed paths. A client that uses the name and keeps hitting those paths is probably not Majestic's crawler, though check the causes of false reports below before concluding that.

How MJ12bot handles robots.txt fetch failures and redirects

Majestic documents its robots.txt handling in a few short lines. Set side by side with RFC 9309, two of them run the other way from the standard.

robots.txt resultMJ12bot, per mj12bot.comRFC 9309
Fetched with rulesObeys the rulesObey the parseable rules
Cannot be retrieved"It will assume that it is okay to crawl your site"Server or network errors (5xx): assume complete disallow
403 ForbiddenTreated as a blanket disallow ("certain failures", 403 named as the example)4xx: the crawler may access any resource
RedirectFollowed, but only within the same siteFollow at least five redirects, even across hosts

What this means in practice:

  • A 403 on robots.txt blocks MJ12bot completely. Most crawlers that follow RFC 9309 read the same 403 as "no rules". If your firewall answers /robots.txt with 403 for bots, MJ12bot stops while others crawl freely. That may be what you want, but it is a side effect, not a rule.
  • An outage or timeout can open your site to MJ12bot. Majestic's wording does not separate 5xx errors from other failures; it only says the bot must be able to retrieve the file. Do not rely on a broken robots.txt to keep MJ12bot out. Majestic's own advice is: "Please always make sure the bot can actually retrieve robots.txt itself."
  • Majestic does not document which other failures count as a blanket disallow, or how 404 or 5xx responses are treated specifically. Serve a real robots.txt with a 200 and the question does not arise.

For how other crawlers read each status, see what crawlers do when robots.txt returns 404, 403 or 500.

Redirects must stay on the same domain

Majestic says MJ12bot follows redirects "but only on the same domain", and that the ideal is robots.txt at /robots.txt as the standard specifies. A setup like this one breaks that:

http
GET /robots.txt HTTP/1.1
Host: www.example.com

HTTP/1.1 301 Moved Permanently
Location: https://www.example-brand.com/robots.txt

Under RFC 9309 a crawler should follow the redirect to www.example-brand.com and apply that file to www.example.com. MJ12bot only follows same-domain redirects, so your rules may never reach it. Serve the file directly on every host instead. Majestic does not define "same domain" further, so whether a redirect between example.com and www.example.com, or between two subdomains, counts is not documented. Test those cases rather than assuming.

Other robots.txt extensions MJ12bot supports

The bot page lists these non-standard extensions:

  • Crawl-Delay up to 20 seconds, in whole seconds; higher values are rounded down to 20.
  • Simple pattern matching in Disallow lines, "compatible with Yahoo's wildcard specification".
  • Allow overriding Disallow when the Allow rule is more specific, meaning longer.

Majestic says a Crawl-Delay in the * group also applies to MJ12bot, but only while there is no MJ12bot group: an MJ12bot group is used instead of the * group, not in addition to it. A Crawl-Delay or Disallow line in * does not reach MJ12bot once it has its own group, so copy the rules you need:

User-agent: MJ12bot
Crawl-delay: 10
Disallow: /search/
Disallow: /cart/

User-agent: *
Disallow: /search/
Disallow: /cart/

Two common complaints have documented answers.

Nofollow links. Majestic says it crawls links marked rel=nofollow. Its explanation is that the attribute was introduced to stop a link influencing the target's PageRank, not to stop a crawler visiting the target, and that a page with several links pointing to it may have nofollow on some and not others. If you want MJ12bot to stay off a page, Disallow it in robots.txt. The attribute is not a crawl control for this bot. rel nofollow, sponsored and ugc covers what the attribute means for Google.

404 and 301 pages. Majestic says it keeps re-crawling them because it has "a long memory" and does not want temporary errors or outages to change a site's profile permanently, and because links to those URLs keep being found. Repeated MJ12bot requests for removed URLs are expected behaviour. If the volume is a problem, return a fast 404 or 410 and consider Crawl-Delay.

Why your robots.txt block seems not to work on MJ12bot

Majestic says it gets reports of violations that turn out to be false positives, and lists three causes. Work through them before you write in or reach for a firewall rule.

  1. robots.txt redirects off the site. Covered above: MJ12bot only follows same-domain redirects, so a cross-host redirect means it never sees your rules.
  2. Several domains logging to one file. If one web server hosts several sites and writes them all to one access log, a request to a site with no block can look like a request to the site that has one. Add the host to each log line, or split logs per domain.
  3. robots.txt tested on the wrong server. Majestic describes complaints where a developer tested against a development server whose robots.txt did not match the live file.

For the second cause, nginx's $host and Apache's %v put the site name on each line. In Apache (the format gets its own name here so it does not replace a vhost_combined format your distribution may already define):

apache
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{User-Agent}i\"" vhost_ua
CustomLog ${APACHE_LOG_DIR}/access.log vhost_ua

For the third, fetch the live file the way a crawler would, from outside your network, and check both the status and the content:

bash
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" \
  -A "Mozilla/5.0 (compatible; MJ12bot)" https://www.example.com/robots.txt
curl -s -A "Mozilla/5.0 (compatible; MJ12bot)" https://www.example.com/robots.txt \
  | grep -i -A5 "mj12bot"

You want 200 with no redirect URL, and an MJ12bot group with the rules you expect. A 403 from this test usually means a firewall or security rule is filtering the user agent, which, as above, MJ12bot reads as a full disallow. Firewall user-agent lists are a common source of surprises; Hetzner's blocked user agents is one example of a host-level block.

If the checks pass and requests to disallowed paths continue, Majestic asks you to email bot@majestic12.co.uk with your site's URL and the log entries showing the bot requesting pages it should not have.

Frequently asked questions

Does Majestic publish IP ranges for MJ12bot?

No. Majestic says that as a community project it cannot restrict the bot to a limited set of IP addresses, and it asks site owners not to block it by IP. It offers a pre-arranged ident string instead for verification.

What is the CRAWLER-IDENT header?

It is a request header in which Majestic can send an ident string you agree with it in advance. Majestic says the string is sent only to the domain or subdomain you ask for, so a request carrying it can be treated as genuine MJ12bot traffic.

What happens if my robots.txt returns 403 to MJ12bot?

Majestic treats certain robots.txt fetch failures, with 403 Forbidden as its example, as a blanket disallow, so MJ12bot will not crawl the site. That is the opposite of RFC 9309, under which a 4xx means no rules apply.

Majestic describes its crawl data as a link graph for its backlinks index and does not mention AI assistants. MJ12bot is not among the crawlers that OpenAI, Anthropic or Perplexity document for their products, so its rules are separate from your AI crawler rules.

Check your robots.txt before you blame the bot

The false violation reports Majestic lists come down to how robots.txt is served and logged: an off-site redirect, shared logs or a stale copy, and a 403 shuts MJ12bot out entirely. The same problems affect AI crawlers in different ways. Run the free check on an important page to see the robots.txt status and the rule that applies to each named AI crawler.

Filed under AI crawlers. Topics: robots-txt, server-logs, rate-limiting, crawl-budget