CCBot and Common Crawl: the open dataset behind many models

CCBot is the crawler run by Common Crawl, a non-profit that collects web pages and publishes them as a free, open archive anyone can download. Because the archive is large, free and easy to process, many AI training datasets start from it. Common Crawl says CCBot follows robots.txt, so a two-line rule stops it from collecting your pages in future crawls, and that rule does not change what AI search crawlers are allowed to fetch.

This guide covers what Common Crawl is, why its data shows up in so much model training, how CCBot behaves, how to block or rate-limit it, and how to tell the real crawler from impostors. Facts about CCBot come from Common Crawl's CCBot page and FAQ, read on 8 November 2026.

What is Common Crawl?

Common Crawl's FAQ describes it as a "501(c)(3) non-profit organization". It crawls the public web and publishes the results for free. Its overview page says the corpus "contains petabytes of data, regularly collected since 2008", and that it is hosted on Amazon Web Services' Public Data Sets and on several academic cloud platforms, with free access to the copy on Amazon.

Each crawl is released as a numbered collection. At the time of writing, the newest listed in Common Crawl's index of collections is CC-MAIN-2026-43, the October 2026 crawl, collected between 5 and 18 October 2026. The ones before it are CC-MAIN-2026-39 (September) and CC-MAIN-2026-34 (August), so recent collections have arrived about once a month. Common Crawl does not publish a fixed release schedule.

The Get Started page describes three file formats:

The WET files are the important part for AI. They are the text of billions of pages, already separated from the HTML, sitting in a public bucket (s3://commoncrawl/ in the AWS us-east-1 region). That makes it one of the easiest web-scale text corpora to obtain.

Why Common Crawl data is used so widely in model training

Collecting the web yourself is expensive: you need crawlers, storage, politeness controls and years of history. Common Crawl has done that work and gives the result away, so researchers and companies often begin with it and then filter, deduplicate and clean it.

Some uses are documented in the papers that introduced them:

• Google's T5 paper, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2019), introduced C4, the "Colossal Clean Crawled Corpus", a cleaned English dataset built from Common Crawl.
• OpenAI's GPT-3 paper, Language Models are Few-Shot Learners (Brown et al., 2020), lists a filtered version of Common Crawl in its table of training datasets.

Beyond published papers like these, you usually cannot tell which models used which crawls. Many model developers do not disclose their training data in detail, and derived datasets are copied and re-mixed. So treat "my pages are in Common Crawl" as meaning "my pages are in a public dataset that many people train on", not as proof that any specific model saw them.

This is also why CCBot belongs in the training column, next to GPTBot and ClaudeBot, rather than with search crawlers such as OAI-SearchBot or PerplexityBot. If the distinction is new to you, training, search and user-triggered crawlers explains why it decides most robots.txt choices.

How CCBot behaves on your site

According to Common Crawl's FAQ and CCBot page:

• User agent. CCBot identifies itself as CCBot/2.0 (https://commoncrawl.org/faq/). The robots.txt product token is CCBot.
• robots.txt. It checks robots.txt before fetching and periodically rechecks whether the file has changed.
• Crawl-delay. It obeys Crawl-delay. The FAQ's example: Crawl-delay: 2 limits it to one request every two seconds.
• Backoff. It slows down when a site returns HTTP 429 or 5xx responses.
• nofollow. It currently honors the nofollow attribute on links embedded in your pages.
• Sitemaps. It uses any Sitemap: line announced in robots.txt.
• No JavaScript. The FAQ says "JavaScript is not executed and Cookies are not used." If your text only appears after client-side rendering, CCBot archives an empty shell, a limitation it shares with many AI crawlers.

Crawl-delay support is worth noting because it is not universal. RFC 9309 does not define it, Google ignores it, and Apple and Amazon say Applebot and Amazonbot do not support it. CCBot is one of the crawlers whose operator says it obeys the directive; see crawl-delay and rate limits for AI crawlers for the full picture.

What blocking CCBot changes, and what it doesn't

Blocking CCBot is a training decision, not a search decision. Here is what it does and does not affect:

So the trade-off is narrow. Blocking keeps your future pages out of one large public dataset that many models and research projects draw on. Allowing it keeps your content in that shared archive, which also feeds non-AI work such as web research and link analysis. Neither choice changes what AI search crawlers are allowed to fetch.

If you care about pages already collected, Common Crawl keeps an Opt-Out Ledger (formerly the Opt-Out Registry) that publishes the legal opt-out requests it has received, and directs questions to info@commoncrawl.org. The post says Common Crawl is "committed to honoring legitimate opt-out requests" but does not describe removing pages from existing archives. If removal matters to you, ask Common Crawl directly rather than assume.

How to block CCBot in robots.txt

Common Crawl's own instruction is a named group:

User-agent: CCBot
Disallow: /

To keep CCBot out of part of the site only:

User-agent: CCBot
Disallow: /members/
Disallow: /drafts/

To keep it but slow it down (Common Crawl's FAQ uses Crawl-delay: 2 as its example, meaning one request every two seconds):

User-agent: CCBot
Crawl-delay: 5

A group like this one contains no Disallow lines, so CCBot may fetch everything on the site, just more slowly. Two rules of robots.txt catch people here. First, a crawler that finds a group naming it follows only that group and ignores User-agent: *, so any general Disallow lines you rely on must be repeated inside the CCBot group. How robots.txt groups work shows this in detail. Second, robots.txt applies per host, so www.example.com, example.com and shop.example.com each need their own file.

A common pattern is to block several training crawlers while leaving search crawlers alone:

User-agent: CCBot
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

That keeps Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot under the general group. The same approach, vendor by vendor, is in how to allow AI search but block AI training.

Platform toggles that already block CCBot

Several platforms add a CCBot rule for you when you turn on their AI setting, so check before editing by hand:

• Cloudflare's managed robots.txt adds User-agent: CCBot with Disallow: / when Bot Preference Sync is on and at least one AI policy is set to block or disallow training (Cloudflare docs).
• Squarespace's "Block known artificial intelligence crawlers" checkbox, under Settings > Crawlers, includes CCBot. It is off by default (Squarespace help).
• Webflow's Traffic control toggles in Site settings > SEO > Indexing include CCBot in the AI bots group, which is allowed by default (Webflow help).
• Wix has no one-click toggle, but its help article on blocking AI crawlers uses CCBot in its example robots.txt rules.

How to check whether CCBot is blocked or visiting

Test your robots.txt

Fetch the live file the way a crawler does and look for a group that names CCBot:

curl -s https://www.example.com/robots.txt | grep -i -A3 "ccbot"

No output means no group names CCBot, so it follows your User-agent: * group, or no rules at all if there is none. Also check the status code with curl -sI: under RFC 9309 a 404 or other 4xx means no restrictions apply, and a 5xx means a compliant crawler must assume everything is disallowed.

Find it in your logs

Search access logs for the token, and count requests per IP:

grep -i "ccbot" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

Finding AI crawler visits in your server logs covers other log formats and CDN logs.

Verify the IP is really Common Crawl

Common Crawl warns that other crawlers falsely identify as CCBot, so do not trust the user agent alone. Real CCBot runs from dedicated ranges published at index.commoncrawl.org/ccbot.json, which lists ipv4Prefix and ipv6Prefix entries. Over IPv4 you can also use forward-confirmed reverse DNS:

host 18.97.14.84
host <hostname-from-the-first-answer>

The first lookup should return a hostname under crawl.commoncrawl.org, and the second should return 18.97.14.84 again.

Reverse DNS is not yet supported over IPv6, so check IPv6 addresses against the JSON file. Fetch the file automatically rather than copying ranges into a firewall rule, because it changes. If an IP claims to be CCBot and fails these checks, it is someone else, and robots.txt will not stop it: that is a job for your firewall or WAF. The same method works for other crawlers, as how to verify a real AI crawler explains.

See whether your pages are already archived

Common Crawl's URL index lets you look up a URL in a given collection. Collection names come from collinfo.json:

curl -s "https://index.commoncrawl.org/CC-MAIN-2026-43-index?url=example.com/*&output=json&limit=5"

Each JSON line is a capture with its timestamp, status and the WARC file that holds it. The index server can be slow and may return a 504 timeout (it did when we tested this query), so keep limit small and retry later.

Frequently asked questions

Does blocking CCBot hurt my SEO or AI search visibility?

No. Google Search, Bing, ChatGPT search, Perplexity and Claude's search each document their own crawler and robots.txt token, and none of them is CCBot. Blocking CCBot only stops your pages being collected into future Common Crawl releases.

Does blocking CCBot remove my site from existing Common Crawl data?

Not according to anything Common Crawl documents. Its FAQ and CCBot page describe blocking future crawls, and copies already made by others are outside its control. For removal questions, Common Crawl points to its Opt-Out Ledger and info@commoncrawl.org.

Does CCBot obey robots.txt and Crawl-delay?

Yes. Common Crawl says CCBot checks robots.txt before fetching, rechecks it periodically, obeys Crawl-delay and slows down on 429 and 5xx responses.

Does CCBot see content rendered by JavaScript?

No. Common Crawl's FAQ says JavaScript is not executed and cookies are not used, so CCBot records only the HTML your server sends.

Is blocking GPTBot enough to keep my content out of OpenAI's training data?

Not necessarily. Disallowing GPTBot covers what OpenAI's own crawler collects, while public datasets such as Common Crawl are gathered by other crawlers, and OpenAI's GPT-3 paper did list filtered Common Crawl data. If you want to opt out of AI training broadly, add CCBot and the other training tokens too, and accept that past crawls and copies are a separate matter.

Check where your site stands

Whether you block CCBot or not, it is easy to block the wrong crawler by accident, for example a shared group that catches search bots too. As a next step, run the free check on one page. It needs no account and shows which of Crawlito's published checks the page passes, including how your robots.txt treats AI crawlers.