Internet Archive vs Common Crawl vs archive.today: Which Web Archive Should You Pull From?

By Nicholas St. Germain — August 30, 2026

Introduction

"Just pull it from the archive" is one of those suggestions that sounds like it saves you a scraping project and usually does not. The three services people mean when they say it, the Internet Archive's Wayback Machine, Common Crawl, and archive.today, are not three flavors of the same thing. They were built by different organizations, for different reasons, with different capture models, and they answer different questions.

Picking the wrong one wastes days. Teams start a historical pricing study against Common Crawl and discover the pages they need were never sampled. Teams build a monitoring tool on the Wayback Machine and discover their target only has four captures across two years. Teams try to automate archive.today and discover there is no API and the front door is behind an interstitial challenge.

This post covers what each archive actually holds, how to query it programmatically, what it will never give you, and how to decide between them in under a minute. At the end we cover the gap all three share, and what fills it.

The One-Paragraph Version

The Wayback Machine is a longitudinal record: many captures of the same URL over time, going back to 1996, queryable by date. Common Crawl is a bulk corpus: a broad monthly sample of the web released as raw WARC files you can process at scale for free. archive.today is an on-demand evidence locker: a human clicks a button, a single page gets frozen with its rendered layout intact, and that snapshot is permanent and citable.

Time series, bulk corpus, evidence. If you know which of those three words describes your project, you already know which archive to open.

Side-by-Side

The Internet Archive and the Wayback Machine

The Internet Archive is a US nonprofit digital library founded in 1996. The Wayback Machine is its web-archiving arm, and it is the only one of the three services designed around the question "what did this exact URL look like on this date."

Its crawls come from several places at once: the Archive's own crawlers, partner and institutional crawls, contributed collections, and Save Page Now submissions from ordinary users. That mixed sourcing is why capture density varies so wildly. A major news homepage may have thousands of captures per year. A mid-size ecommerce product page may have three, all from whenever someone happened to link to it.

Querying It: The CDX Server

The CDX Server is the API worth learning. It queries the same index Wayback uses internally and returns one row per capture.

curl -s 'https://web.archive.org/cdx/search/cdx?url=example.com/pricing&output=json&from=2024&to=2026&filter=statuscode:200&collapse=digest'

The parameters that matter in practice:

• matchType set to exact, prefix, host, or domain. Use domain to sweep subdomains, or append /* to the URL as shorthand for a prefix query.
• from and to take 1 to 14 digits of yyyyMMddhhmmss, so from=2024 is valid.
• filter applies a regex to any field and inverts with a leading !. Filtering to statuscode:200 alone removes a surprising amount of noise.
• collapse=digest is the single most useful parameter here. The digest is a content hash, so collapsing on it removes adjacent captures where nothing changed. For a change-detection job this turns 400 rows into the 11 that represent actual edits.
• collapse=timestamp:6 gives you at most one capture per month instead.
• limit takes a negative value to read from the newest end backwards.
• showResumeKey and resumeKey page through large result sets without offset drift.

Default output fields are urlkey, timestamp, original, mimetype, statuscode, digest, and length. Narrow them with fl when you are pulling a lot of rows.

Fetching the Bytes, Not the Banner

A normal Wayback URL serves you the capture wrapped in the Archive's navigation banner, with links and asset paths rewritten to point back into the archive. For parsing, that rewriting is a liability. Append id_ to the timestamp to get the original response body as captured:

# Rewritten replay, good for humans
https://web.archive.org/web/20260115120000/https://example.com/pricing

# Original bytes, good for parsers
https://web.archive.org/web/20260115120000id_/https://example.com/pricing

There is also a lightweight Availability API for the single common case of "is there any capture near this date":

curl -s 'https://archive.org/wayback/available?url=example.com/pricing&timestamp=20250601'

It returns an archived_snapshots.closest object with url, timestamp, status, and available, or an empty archived_snapshots object when nothing exists. It is the right tool for a 404 handler and the wrong tool for anything that needs more than one result.

Where It Falls Down

Capture density is not something you control retroactively. If nobody archived the page during the window you care about, no API parameter conjures it.

Replay fidelity degrades on JavaScript-heavy pages. The Wayback Machine captures the assets it can see and rewrites URLs at replay time, but a single-page app that assembles itself from XHR calls to endpoints that no longer exist will render as an empty shell. The capture is technically there; the content is not.

And retroactive exclusions happen. Pages can disappear from replay after the fact for policy or legal reasons, which is exactly the property that makes archive.today attractive to some users.

Common Crawl

Common Crawl is a different animal entirely. It is a nonprofit that has been publishing an open crawl of the web since 2008, and it releases a fresh crawl archive roughly every month under an identifier like CC-MAIN-2026-34. The August 2026 archive holds about 2.14 billion pages. January 2026 was larger at roughly 2.3 billion pages, about 398 TiB uncompressed, spanning 44.9 million hosts and 36.9 million registered domains, including 616 million URLs never seen in any earlier crawl.

Nobody queries this a page at a time. It is a corpus, and it comes in three parallel formats:

Each of those is split across 100,000 files per crawl, plus separate collections for robots.txt fetches and non-200 responses. If you want a link graph, WAT is a fraction of the download. If you are training or fine-tuning on text, WET is smaller still. Reach for WARC only when you need the actual markup.

Two Ways In

For a handful of lookups, the index API is a plain HTTP call:

curl -s 'https://index.commoncrawl.org/CC-MAIN-2026-34-index?url=example.com%2F*&output=json' | head

Each JSON line carries a filename, offset, and length. Those three values are a byte range into a WARC file, so you can pull one page out of an 84 TiB corpus with a range request:

import gzip, io, requests

rec = {"filename": "crawl-data/CC-MAIN-2026-34/segments/.../warc/CC-MAIN-....warc.gz",
       "offset": 123456789, "length": 45678}

start = int(rec["offset"])
end = start + int(rec["length"]) - 1
r = requests.get(
    "https://data.commoncrawl.org/" + rec["filename"],
    headers={"Range": f"bytes={start}-{end}"},
    timeout=60,
)
raw = gzip.GzipFile(fileobj=io.BytesIO(r.content)).read()
# raw now holds the WARC record: WARC headers, HTTP headers, then the body

For anything bigger, use the columnar index instead. It ships as Parquet alongside each crawl (900 files for CC-MAIN-2026-34, about 0.20 TiB) and is meant to be queried with Athena or Spark. Filtering to one TLD, one content type, or one status code across a whole crawl is a SQL query there and an all-day download otherwise. Run it in us-east-1 where the bucket lives so you are not paying to drag terabytes across regions.

The Honest Limits

Coverage is a sample, not a census. Two billion pages sounds like everything and is not. Common Crawl selects what to fetch and does not fetch your target site exhaustively. Deep catalog pages, paginated result sets, and anything gated behind query parameters are routinely absent. Check the index before you plan around it.

It is HTML at fetch time, not rendered DOM. CCBot does not run JavaScript. Client-rendered content simply is not in the corpus.

CCBot honors robots.txt. It identifies as CCBot/2.0 (https://commoncrawl.org/faq/), obeys Crawl-delay, backs off on 429 and 5xx responses, and can be blocked outright with a two-line robots rule. Since the AI training debate got loud, a lot of large publishers have added exactly that rule. Their recent pages are gone from the corpus even though older crawls still contain them.

It is always behind. A crawl released in late August reflects fetches from earlier in the month. For a price, a stock level, or a ranking, weeks-old is the same as wrong.

archive.today

archive.today, reachable through mirrors including archive.is, archive.ph, archive.li, archive.vn, archive.fo, and archive.md, has been running since 2012 under an operator who has never publicly identified themselves. It is the smallest of the three in scope and the most useful for one narrow job: freezing a specific page, right now, in a form you can cite later.

Submit a URL and it stores two things: a snapshot that reproduces the page as rendered, and a screenshot. Because it renders, it handles a lot of the JavaScript-heavy pages the Wayback Machine flattens. Because it explicitly does not follow robots.txt, reasoning that it acts as a direct agent of the person who asked, it captures pages other archives skip. And once a snapshot exists, ordinary users cannot delete it, which is the property journalists and researchers actually want.

The tradeoffs are just as sharp:

• There is no public API. No CDX equivalent, no bulk export, no documented query interface.
• The front door is defended. Automated access runs into challenge pages, and mirror domains have a long history of DNS resolution problems with some public resolvers.
• Coverage is whatever humans requested. There is no crawl behind it, so there is no such thing as a systematic sweep of a domain.
• It stores text and images. Non-static content is out of scope.
• The governance is opaque. For a service whose entire value proposition is durability, an anonymous operator with no published funding model is a real risk to weigh.

Treat archive.today as a citation tool used by hand, not as a data source you build a pipeline on.

How to Choose in 30 Seconds

Answer the first question that applies:

1. Do I need the same URL at many points in time? Wayback Machine. Nothing else does time series.
2. Do I need millions of pages across many domains, and I do not care which specific ones? Common Crawl. It is free bulk data and downloading it hits no one's origin server.
3. Do I need to prove this exact page said this exact thing today? archive.today, with a Save Page Now submission to the Wayback Machine as a second copy.
4. Do I need this specific page's current state, or every page on a site, or content that only appears for a real browser? None of the three. That is a live fetch.

That fourth case is the one that surprises people, so it is worth being blunt about the boundary.

What No Archive Can Give You

Every archive shares four structural gaps, and they are the reason archive-first projects so often turn into scraping projects halfway through.

Freshness. The newest thing in Common Crawl is weeks old. The newest Wayback capture of your target might be from 2023. Any workload keyed on current state, pricing, inventory, availability, rankings, is out of scope by construction.

Completeness on a chosen target. Archives capture what their crawl policy and their users happened to reach. When you need all 40,000 product pages on one site, no archive has all 40,000.

Rendered, interaction-dependent content. Common Crawl does not render at all. Wayback replay of app-shell pages frequently comes back empty. archive.today renders, but one page at a time, by hand.

Geographic and session variation. This one is quietly the biggest. Every archive captures from one vantage point. CCBot fetches from published cloud IP ranges, and the Internet Archive crawls from its own infrastructure. A page that shows different prices, different availability, different currency, or different search results depending on where the request comes from is represented in the archive by exactly one of those variants. If your question is "what does this look like to a US shopper," an archive cannot answer it, no matter how many captures it has.

There is also a politeness point worth stating plainly: these archives are free public infrastructure run by nonprofits. Their APIs are rate limited, and pounding them is both rude and self-defeating. If your volume needs are large enough that rate limits are a problem, that is a signal you should be fetching from origin, not from the archive.

The Practical Pattern: Archive First, Live Fetch for the Gaps

The workflow that actually holds up in production uses archives for what they are good at and live requests for the rest.

import requests

WAYBACK_CDX = "https://web.archive.org/cdx/search/cdx"

PROXY = "http://USERNAME:PASSWORD@isp.statproxies.com:3128"
PROXIES = {"http": PROXY, "https": PROXY}


def archived_versions(url, since="2024"):
    """Historical captures, free, no load on the target site."""
    r = requests.get(
        WAYBACK_CDX,
        params={
            "url": url,
            "output": "json",
            "from": since,
            "filter": "statuscode:200",
            "collapse": "digest",
            "fl": "timestamp,digest,length",
        },
        timeout=30,
    )
    rows = r.json()
    return rows[1:] if rows else []  # first row is the header


def current_version(url):
    """Current state, rendered by the origin, from a US IP."""
    r = requests.get(url, proxies=PROXIES, timeout=30)
    r.raise_for_status()
    return r.text


history = archived_versions("example.com/pricing")
if not history:
    print("no captures, this dataset has to be built from live fetches")

today = current_version("https://example.com/pricing")

The split is the point. Backfill comes from the archive at zero cost and zero load on anyone's origin. The current row, the pages the archive never captured, and anything that varies by location come from a live request through your own IPs.

For that live half, IP stability matters more than most people expect. A price-history table is worthless if half the rows were collected from a German exit node and half from a US one, because you are no longer measuring the same thing over time. That is the argument for static IPs over rotating pools on this particular workload: our static US ISP proxies keep the same address across runs, so week 12 is comparable to week 1. Rotation is genuinely better for hostile anti-bot targets that burn individual IPs, and we do not offer it, which is worth knowing before you pick us for that job. Our related write-up on datacenter vs residential vs ISP proxies covers where each type wins.

Cost, Honestly

All three archives are free to query. The costs are elsewhere:

The Common Crawl line is the one that bites teams. The data is free and the bill is not, if you download 84 TiB to a machine in the wrong region instead of filtering with Athena where the bucket already lives.

FAQ

What is the difference between the Internet Archive and Common Crawl?

The Internet Archive runs the Wayback Machine, which stores many captures of the same URL over time and lets you retrieve any specific date through its CDX and Availability APIs. Common Crawl publishes a broad monthly sample of the web as bulk WARC, WAT, and WET files on S3, designed for large-scale processing rather than per-URL lookups. Use the Wayback Machine for history of one page, Common Crawl for volume across many pages.

Is Common Crawl a complete copy of the internet?

No. Each monthly crawl covers roughly 2.1 to 2.3 billion pages, which is a broad sample rather than a census. Deep catalog pages, paginated results, and URLs behind query parameters are frequently missing, CCBot does not execute JavaScript, and any site that blocks CCBot in robots.txt is excluded. Always check the index for your specific URLs before planning a project around it.

Can I use the Wayback Machine API for web scraping?

You can query the CDX Server API for historical captures and fetch the original bytes of any capture by appending id_ to the timestamp in the replay URL. It works well for backfilling history and repairing dead links. It is not a substitute for scraping current data: captures may be sparse or years old, JavaScript-heavy pages often replay empty, and the API is rate limited free infrastructure that should not be hammered.

Does archive.today have an API?

No. archive.today has no public API, no bulk export, and no documented query interface, and automated access typically runs into anti-bot challenges. It is designed for a person to save and cite one page at a time. For programmatic archive access, use the Wayback Machine's CDX Server or Common Crawl's index.

Why do archived pages show different prices than the live site?

Because every archive captures from a single vantage point at a single moment. Prices, currency, availability, and search results that vary by visitor location are frozen as whatever the crawler saw from its own IP, usually a cloud or archive-owned address in one country. To see what a specific market's visitor sees, you have to request the page live from an IP in that market.

Which archive should I use to prove a page said something?

archive.today is the strongest option for that specific job, because it renders the page, ignores robots.txt, and does not let ordinary users delete snapshots. Submit the same URL to the Wayback Machine's Save Page Now as a second, institutionally backed copy. Keeping two independent snapshots protects the citation if either service changes its policy.

Conclusion

The three archives are not competitors, they are different instruments. The Wayback Machine answers "what did this page look like then," and its CDX API with digest collapsing is the fastest way to find real changes over time. Common Crawl answers "give me a lot of the web cheaply," and the columnar index is how you use it without a five-figure compute bill. archive.today answers "make this page permanent and citable right now," and it does that job by hand, no API involved.

None of them answers "what does this page look like today, to a visitor in this country, with JavaScript running." That gap is not a flaw in the archives. It is the definition of the boundary between archival research and live data collection, and pretending it does not exist is what turns a two-week project into a two-month one.

Build the historical layer from archives. Build the current layer yourself. If the current layer needs stable US IPs that stay identical run over run, that is what we sell, from $2.50 per IP per month, no bandwidth metering.

Crawl statistics cited from Common Crawl's published archive listings for CC-MAIN-2026-34 (August 2026) and for the January 2026 crawl. Wayback Machine scale reflects the Internet Archive's one trillion page captures milestone announced October 22, 2025.