How to find your URL in a Common Crawl WARC record
Look up a page in the Common Crawl URL Index, retrieve one archived response by byte range, and compare it with the HTML your site serves today.
By Crawlito · 5 min read
In short
- The URL Index can identify captures in a specific Common Crawl collection.
- Filename, offset, and length locate a compressed record inside a WARC file.
- A capture documents one historical fetch and cannot establish whether a particular model used the page.
On this page
To see whether Common Crawl captured a particular page, search its URL Index for the exact public URL in a chosen collection. A matching record gives you a capture time and the information needed to retrieve the archived response. Comparing that capture with your current HTML can reveal when a page served a login screen, thin shell, or obsolete version. Common Crawl URL Index
Start with an exact URL and a meaningful date
Choose a page with an identifiable fact you can search for later, such as a manual with a distinctive part number. Record its current URL, the address in use during the period you care about, and any hostname migration date. The latest collection is a sensible first look for a recent page. For a page that moved last year, use a collection from before or after the move depending on which deployment you are investigating.
Open the index interface, select the collection, and submit the URL. Save the collection ID alongside the result. The service also lists collections in machine-readable form and links its CDX query API documentation. For a small investigation, use exact queries and a short list of plausible variants. Common Crawl asks users to avoid bulk scraping its public index and offers other data access methods for large analyses.
A lookup can return several captures for related addresses, including redirects or equivalent-looking paths. Compare the original URL, timestamp, HTTP status, and MIME type on each candidate. An archived permanent redirect is evidence about the old address. Its body cannot show what the destination served unless you find the destination's own record. The guide to CCBot provides background on the crawler and its role.
Locate the record you intend to read
In the JSON result, look for filename, offset, and length. The filename identifies a compressed archive object and the other fields locate the record inside it. Use the exact values from one record together. Combining an offset from one capture with a filename from another can retrieve unrelated bytes or fail decompression. Keep the original index line as a local JSON file so the process is repeatable.
Common Crawl documents WARC as its raw capture format. WAT contains derived metadata and WET contains extracted text. If you want to know what HTTP status or content type was delivered, a response record in WARC is the useful starting point. An extracted text file has already passed through additional processing. Common Crawl’s data formats and access examples
The byte interval is inclusive. If a record begins at offset 100 and has length 25, the final requested byte is 124. This arithmetic example describes how ranges work rather than a real capture. The following Python example takes a single JSON index record saved as capture.json and checks for a partial response before reading the payload. It expects the ordinary public data host and requires a filename from the selected index record.
import gzip
import json
from pathlib import Path
from urllib.request import Request, urlopen
record = json.loads(Path("capture.json").read_text())
start = int(record["offset"])
size = int(record["length"])
archive = "https://data.commoncrawl.org/" + record["filename"]
request = Request(archive, headers={"Range": f"bytes={start}-{start + size - 1}"})
with urlopen(request, timeout=30) as response:
if response.status != 206:
raise RuntimeError("Archive server did not honor the byte range")
compressed = response.read(size)
if len(compressed) != size:
raise RuntimeError("Incomplete record")
Path("capture.warc").write_bytes(gzip.decompress(compressed))Read the archived response in context
Open the resulting record locally as data. WARC headers describe the capture, and an ordinary response record then contains the HTTP response headers and payload received from the site. Note the target URI, WARC date, record type, HTTP status, and content type before searching for your chosen text. A revisit record can refer to an earlier payload, and a truncated capture can omit later content. Record these conditions and choose a complete response when you need a direct body comparison.
Look for a distinctive heading in a guide or a part number near the main answer. A 200 status with only a loading shell is different evidence from a 200 with a full article. A 403 with a security challenge identifies the response delivered for that capture, without establishing what happened to every other crawler. The raw versus rendered HTML guide explains why a browser screenshot today can show more than the historical response contains.
For a meaningful comparison, fetch the current public article without a signed-in account and save the current date and status. Compare the same markers in each copy. If the archive predates the article's launch, a missing marker is expected. If the old record contains a challenge while the current public page contains the article, document those two observations and the change date if known. Historical absence alone cannot diagnose today's access rule.
Interpret a blank lookup conservatively
A collection can lack the page you want. Try the previous hostname, the known canonical, or an adjacent collection that covers the relevant time. Check whether a former URL had a trailing slash, a language path, or a redirect you have since removed. Keep a list of what you looked up so a negative result can be reported accurately. “No match for these two addresses in these collections” is an observation another person can reproduce.
If you also have server logs, compare verified historical crawler requests with the dates of your lookups. A logged request and a searchable archive record are different pieces of evidence. A public record demonstrates capture in that dataset. Establishing that any given model, product, or answer used the page would require separate evidence about that system. The crawler traffic measurement guide helps define the observation available from logs.
Tell an archived challenge from an archived article
Imagine an index row that lists the right URL, a success status, and an HTML MIME type. After extracting the response, its title reads “Please enable cookies” and the body lacks a prominent phrase from the article. Inspect a short stretch of ordinary body text around the expected article heading. A copy of the title inside a script or a return URL can put a familiar keyword in an otherwise empty challenge. Treat the response as a page a person could actually read when evaluating whether the article was present.
Now compare with a capture from another collection, if one exists. Check the same markers and note each timestamp. Two different payloads at one URL can document when the serving behavior changed between observations, but they do not tell you the precise hour the change happened. Use a deployment record or retained server logs to narrow that interval if your team needs a more precise explanation.
If a binary payload or a compressed body cannot be read as plain text, examine the archived HTTP headers and encoding before declaring the article missing. A local WARC-aware reader can handle formats that a text search misses. Keep the extracted file local rather than opening untrusted archived HTML as an active webpage when all you need is its historical content.
Frequently asked questions
Why does the WARC have an old page title?
It preserves the response from the recorded time. Check the WARC date, target URI, and the site's publication history before interpreting the difference.
Is one index query enough to rule out every capture?
No. Collections and URLs vary. Report the exact addresses and collection IDs you checked.
Can I use this procedure for thousands of domains?
The public index is meant for limited queries. Common Crawl points researchers with bulk filtering needs toward its columnar index and downloadable datasets.
Compare with the page you serve now
Run the free page check on the current public URL. Put the live findings next to the dated WARC record to see whether the specific delivery issue still exists.
Filed under AI crawlers. Topics: ccbot, common-crawl, raw-html, troubleshooting