Skip to content

archive.org_bot: block or allow the Wayback Machine crawler

What archive.org_bot is, how Internet Archive and Archive-It crawlers treat robots.txt, and how to ask for existing Wayback Machine snapshots to be excluded.

By Crawlito · 10 min read

In short

  • Archive-It documents archive.org_bot as the user agent of its crawlers, and says they generally include archive.org_bot or Archive-It in the user agent string.
  • Archive-It's Standard crawler respects robots.txt exclusions and crawl delays by default, but subscribers can add an Ignore Robots.txt scoping rule for a seed or host.
  • The Internet Archive said in 2017 that robots.txt files written for search engines do not work well for web archives, and that it had stopped honoring them on US government and military sites.
  • robots.txt is not a documented way to remove existing captures; the documented route is an email to info@archive.org, and the Internet Archive guarantees no outcome.
  • archive.org_bot is not an AI search crawler, so blocking it does not change what Googlebot, OAI-SearchBot, Claude-SearchBot or PerplexityBot may fetch.
On this page

archive.org_bot is the user agent the Internet Archive's Archive-It service names for its crawlers, the ones that capture pages for web archive collections that also appear in the Wayback Machine. You can block it with a robots.txt group that names archive.org_bot, and Archive-It's standard crawler respects that by default. Two limits matter before you decide: a block is aimed at future crawls and is not a documented way to remove snapshots already captured, and the Internet Archive has said robots.txt is not the rule it relies on for every archive crawl.

This post covers how to identify the crawler, what the Internet Archive and Archive-It actually document about robots.txt, how to block or allow it, how to see what is already archived, and how to ask for removal. It does not tell you which choice to make: that depends on how you weigh a public historical record against reuse of your pages by others.

What archive.org_bot is, and what ia_archiver was

The Internet Archive runs two web archiving services that share infrastructure:

ServiceWhat it isWho chooses what gets crawled
General Archive (web.archive.org)The free Wayback MachineThe Internet Archive's own crawls (its help page also names Alexa Internet's crawls), plus pages people save with Save Page Now
Archive-ItA fee-based subscription service offered by the Internet Archive to institutionsThe subscribing organization, which sets seeds, scope and crawl frequency

Archive-It's article on the difference between the two says that, in general, sites collected and made public on Archive-It also appear in the General Archive. So a crawl started by a university library can end up viewable in the Wayback Machine.

On the crawler name, Archive-It's troubleshooting article states it plainly: the user agent of its crawler is archive.org_bot. Its page on custom user agents adds that its crawlers "generally include archive.org_bot or Archive-It" in the user agent string, and that a partner organization can ask for a custom user agent for its whole account or a single collection. That is why a robots.txt group or a server rule should account for both strings.

ia_archiver is the older token you will still see in many robots.txt files and blocking guides. The Internet Archive's current help pages and Archive-It's documentation do not mention it, and the Wayback Machine help page still mentions Alexa Internet when it describes how sites are discovered. Neither organization documents today whether any current Internet Archive crawler reads a group addressed to ia_archiver. If you want to be thorough, name both tokens; if you only add one, archive.org_bot is the one the Internet Archive itself documents.

For identification beyond the user agent, Archive-It does not publish its IP ranges in its help center. It tells partners to submit a ticket to get the IP range to share with a site owner. Without a published list or documented reverse DNS, treat log entries that claim to be archive.org_bot with the same caution as any other unverified bot: the user agent alone proves nothing.

How the Internet Archive treats robots.txt

This is where the Internet Archive differs from most crawler operators, and from Common Crawl, which simply states that CCBot obeys robots.txt (see CCBot and Common Crawl).

Archive-It crawls

Archive-It's article on robots.txt exclusions gives three documented behaviors:

  • Standard crawls (the Heritrix crawler plus Umbra, a browser-based tool, as Archive-It's crawling technology article describes) respect all robots.txt exclusions and crawl delays by default.
  • Brozzler crawls, its browser-based crawler, respect most exclusions but ignore exclusions on embedded documents such as images and style sheets, and ignore crawl delays, by default.
  • Subscribers can override robots.txt. If contacting the site owner is not an option, a partner can add an "Ignore Robots.txt" scoping rule to a seed or host, which tells the crawler to collect in-scope documents anyway.

Archive-It's own example of allowing its crawler while blocking everyone else looks like this:

User-agent: archive.org_bot
Disallow:

User-agent: *
Disallow: /

So a block is a default that Archive-It honors, not a guarantee. Archive-It suggests that a partner who wants your site contact you if that is an option, and points to the Ignore Robots.txt rule when it is not.

General Archive crawls

The Internet Archive's own crawls are less clearly documented. In April 2017, Wayback Machine director Mark Graham wrote Robots.txt meant for search engines don't work well for web archives. The post said the Archive had stopped referring to robots.txt files on US government and military websites, for both crawling and displaying captures, and that it saw "the future of web archiving relying less on robots.txt file declarations geared toward search engines".

The post also explained the problem it was reacting to. When a site became a parked domain and the new robots.txt blocked everything, that had historically removed the whole domain's history from view in the Wayback Machine.

What the Internet Archive has not published since is a crawler policy page that says, for its own crawls in 2026, which robots.txt groups it reads and when it ignores them. The current Wayback Machine help page still lists robots.txt as one reason a site may not have been crawled, says some sites "may have been excluded from the Wayback Machine due to a robots.txt file on the site", and asks site owners who want to be archived to check that robots.txt and meta robots rules do not turn crawlers away. That suggests robots.txt still matters, but it does not tell you exactly how.

How to block or allow archive.org_bot

Block it in robots.txt

User-agent: archive.org_bot
User-agent: ia_archiver
Disallow: /

To keep it out of a section only:

User-agent: archive.org_bot
Disallow: /members/
Disallow: /account/

Remember that under RFC 9309 a crawler which finds a group naming it follows only that group and ignores User-agent: *. If your general group has rules you rely on, repeat them inside this one. How robots.txt groups work shows the pattern, and each host (www, the bare domain, subdomains) needs its own file.

Block it at the server

robots.txt is a request. Because Archive-It partners can choose to ignore it, a site that needs an actual refusal has to do it at the server or edge. In Nginx, inside the server block, this refuses matching user agents everywhere except /robots.txt:

nginx
set $block_archive 0;
if ($http_user_agent ~* "(archive\.org_bot|archive-it|ia_archiver)") {
    set $block_archive 1;
}
if ($uri = /robots.txt) {
    set $block_archive 0;
}
if ($block_archive) {
    return 403;
}

The exception matters. Under RFC 9309, a crawler that gets a 4xx response for robots.txt may treat the file as unavailable and crawl without restrictions, so a 403 on the file itself would hide the group you wrote. This rule only catches requests that identify themselves. Blocking crawlers in Nginx, Apache and .htaccess has the Apache and .htaccess versions. Keep the robots.txt group too, so a crawler that does read the file gets the same answer without spending a request on a 403.

Allow it explicitly

If your general group is restrictive and you want archiving, give the crawler its own group with an empty Disallow, as in Archive-It's example above. If you use a crawl delay, note that Brozzler ignores it by default.

What blocking does not touch

Blocking archive.org_bot affectsEffect
Future Archive-It Standard crawlsStopped by default; a partner can add an Ignore Robots.txt rule
Future Brozzler crawlsMostly stopped; embedded files may still be fetched
Captures already in the Wayback MachineNot documented today: a later block once hid a parked domain's history, but no current page says what a new block does
Pages saved through Save Page NowNot documented whether it reads your robots.txt
Google Search, AI Overviews, ChatGPT search, Perplexity, Claude searchNo effect: each is controlled by its own crawler token

Why blocking is not a removal request

A robots.txt rule speaks to crawlers about future requests. The captures already in the Wayback Machine were made earlier and sit in the archive's own storage. The 2017 post shows that a later robots.txt block did, in the past, hide a parked domain's earlier history from view, and the help page still says robots.txt can be a reason a site is excluded. Neither page says whether adding a block today hides existing captures, so do not rely on it. The Internet Archive's documented route for removal is separate:

  1. Email info@archive.org.
  2. List the URLs you want excluded, the time period, and your period of control over the site or account.
  3. Add anything else that helps the review.

The help page says the request is reviewed by its team and that it does "not make any guarantees beforehand about the outcome of a request". Plan for the possibility of a no, and keep your own record of what you asked for and when.

Two other things are outside the Internet Archive's control entirely: copies that third parties have already downloaded from the Wayback Machine, and references to archived URLs in other sites. If the goal is to get a page out of AI answers rather than out of the archive, that is a different set of controls; see how to remove a page from AI search.

How to check what the Wayback Machine already holds

Before deciding, look at what is there. The Internet Archive documents two public APIs.

The Wayback Availability API returns the capture closest to a date:

bash
curl -s "https://archive.org/wayback/available?url=example.com/pricing&timestamp=20250101"

An empty archived_snapshots object means no capture is available. Otherwise closest gives the snapshot URL, its timestamp and the HTTP status that was captured.

The CDX server lists captures, with filtering:

bash
curl -s "https://web.archive.org/cdx/search/cdx?url=example.com/blog/&matchType=prefix&output=json&limit=20&fl=timestamp,original,statuscode"

With output=json, the first row is the field names. matchType=prefix returns everything under a path, from and to narrow the date range, and filter=!statuscode:200 shows captures that were not successful responses. Use the list to decide which URLs and time periods to name in a removal request. The help page notes a 3 to 10 hour lag between a crawl and its appearance in the Wayback Machine, so a very recent visit in your logs may not show up yet.

To find visits in your own logs:

bash
grep -iE "archive\.org_bot|archive-it|ia_archiver" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

Match both strings, because partner crawls with a custom user agent may carry Archive-It rather than archive.org_bot.

The trade-off: archival value against third-party reuse

The case for allowing it is the one the Internet Archive makes: a public record of what a site said and when. Other people can link to an archived copy to show what a page said on a given date, and your own team may need an old version of a pricing page, policy or press release that you no longer have. Blocking also does not affect search: archive.org_bot is not a search or AI search crawler, so it plays no part in whether Google or an AI assistant can read your live pages.

The case for blocking is about reuse. Wayback Machine captures are public, so anyone can read or copy them, including people who are blocked from your live site. In early 2026 several large publishers were reported to have started restricting the Internet Archive's access for that reason, citing concern that the archive could be used to collect their content for AI systems. The Internet Archive's response, in a February 2026 post, quoted Wayback Machine director Mark Graham calling those concerns "understandable, but unfounded" and saying the archive works to prevent such abuse. In September 2026 it described protections against high-volume automated traffic, including blocking with HTTP 429 responses. Neither post sets out a public technical specification, so you cannot verify from documentation alone how well bulk collection is prevented.

A middle path some sites take is to allow archiving of public, stable pages and disallow sections that change often or hold members' content. Whatever you choose, make it match your other training decisions; should you block AI crawlers sets out the wider framework, and do AI crawlers respect robots.txt covers why a robots.txt line is only as strong as the crawler reading it.

Frequently asked questions

Does blocking archive.org_bot hurt SEO or AI search visibility?

No. Google, Bing, ChatGPT search, Perplexity and Claude's search each use their own documented crawlers. archive.org_bot only affects what web archives collect.

Will blocking archive.org_bot delete my old Wayback Machine snapshots?

The Internet Archive does not currently document what a new block does to existing captures, although before 2017 a later block could hide a parked domain's history. To ask for captures to be excluded, email info@archive.org with the URLs, the time period and your period of control over the site; the outcome is not guaranteed.

Should I block ia_archiver or archive.org_bot?

archive.org_bot is the token Archive-It documents. ia_archiver is a legacy token that current Internet Archive help pages do not mention, so naming both in one group is the cautious option.

Can someone archive my page with Save Page Now if I block the crawler?

Save Page Now lets anyone save a single page once. The Internet Archive does not document whether Save Page Now checks your robots.txt, so if you need a hard refusal, block at the server as well.

Check which crawlers your rules actually reach

A group added for archive.org_bot is easy to get wrong in ways that also catch crawlers you meant to allow. After changing robots.txt or server rules, run the free check on an important page to confirm the main AI search crawlers can still fetch it.

Filed under AI crawlers. Topics: ai-crawlers, robots-txt, server-logs, strategy