Wayback Machine & Common Crawl#

Someone already crawled the web for you. Get the page — and its entire history — without sending the target a single request.

⏱ ~9 min read · ~15 min hands-on 🔗 needs: Legal & Ethical Scraping · Hidden JSON APIs

Two public archives cover a large share of the web: the Internet Archive’s Wayback Machine (snapshots of individual URLs over time) and Common Crawl (petabyte-scale crawls released free for research). Reach for them when a site blocks you, when you need history rather than the current page, or when you need breadth no polite scraper could achieve.

Try it in 5 minutes — a URL’s whole history#

The Wayback CDX API lists every capture it holds:

curl -s 'https://web.archive.org/cdx/search/cdx?url=iitm.ac.in&output=json&limit=5'

You get a JSON array whose first row is the column headers — urlkey, timestamp, original, mimetype, statuscode, digest, length. Each later row is one snapshot; the 14-digit timestamp (YYYYMMDDhhmmss) rebuilds the archived URL:

https://web.archive.org/web/20240115120000/https://iitm.ac.in/
# /// script
# requires-python = ">=3.12"
# dependencies = ["httpx>=0.28"]
# ///
"""Count Wayback snapshots per year for a URL — a free time-series, zero load on the site.

Run:  uv run wayback_history.py iitm.ac.in
"""

import sys
from collections import Counter

import httpx

url = sys.argv[1] if len(sys.argv) > 1 else "iitm.ac.in"

rows = httpx.get(
    "https://web.archive.org/cdx/search/cdx",
    params={"url": url, "output": "json", "fl": "timestamp,statuscode", "limit": 2000},
    timeout=60,
).raise_for_status().json()

header, *captures = rows  # first row is the header
per_year = Counter(ts[:4] for ts, _status in captures)
for year, n in sorted(per_year.items()):
    print(f"{year}  {'█' * (n * 40 // max(per_year.values()))} {n}")
print(f"\n{len(captures)} snapshots total")

✅ A histogram of how often a site was archived — built entirely from the archive. The site itself saw nothing.

Which archive, when#

Wayback MachineCommon Crawl
ShapeMany snapshots of one URL over timeOne massive snapshot of many URLs per crawl
Best forHistory, deleted pages, “what did this say in 2019?”Web-scale corpora, cross-site analysis, LLM datasets
AccessCDX API + Availability APIIndex API + WARC files on S3
FreshnessContinuous, uneven per siteMonthly crawls (e.g. CC-MAIN-2026-30)

Availability API — the quick “is there a snapshot near this date?” lookup:

curl -s 'https://archive.org/wayback/available?url=iitm.ac.in&timestamp=20200101'

Common Crawl index — ask which crawls contain a URL pattern. Collections are named by crawl (CC-MAIN-2026-30 was July 2026); the current list lives at collinfo.json:

curl -s 'https://index.commoncrawl.org/CC-MAIN-2026-30-index?url=iitm.ac.in/*&output=json' | head -3

Each line is a JSON record pointing into a WARC file (the raw HTTP request/response, stored on S3 with byte offsets) — so you can fetch just the bytes for one page instead of downloading a petabyte.

⚖️ Archives are a legitimate source, but the content in them still has an owner. Copyright and personal-data rules apply exactly as they would on the live site — see Legal & Ethical Scraping. Be gentle with these APIs too: they’re free public infrastructure.

When it fails#

SymptomCauseFix
Empty CDX resultURL never archived, or wrong formTry www./no-www, add matchType=domain or a * wildcard
Snapshot renders brokenAssets weren’t capturedUse the raw capture: add id_ after the timestamp (/web/20240115120000id_/…)
CDX request times outAsking for a huge domain-wide rangeAdd limit=, narrow with from=/to= years
Common Crawl returns nothingThat crawl didn’t include the URLTry another collection from collinfo.json
Archived page is staleLast capture is years oldConfirm the date — an archive proves was, not is

Your turn (≈15 min)#

  1. Run wayback_history.py on your institution’s domain; note the busiest year.
  2. Use the Availability API to find the closest snapshot to 20200101, open it, and compare against the live site.
  3. Fetch one archived page’s raw HTML (id_ form) and extract its title.
  4. Query the Common Crawl index for a domain you like and count how many URLs it holds.

Checklist#

  • I know the CDX API’s first row is the header row.
  • I can rebuild an archived URL from a 14-digit timestamp.
  • I know when to use Wayback (history of one URL) vs Common Crawl (breadth).
  • I check archives before fighting a site that blocks me.
  • I remember an archived page proves what was true, not what is.

Go deeper#