Archive.org Download Formats: WARC, CDX & Single Files

You found the crawl, hit download, and now you are staring at a 900 MB file ending in.warc.gz that nothing on your machine opens. Welcome to the least documented part of Archive.org: the formats. The library hands out raw capture data, and raw capture data expects you to bring your own tools.

Three formats cover almost everything you will ever pull: WARC files, JSON CDX output, and plain single files. Here is what each one really is, when you want it, and how to turn it into something useful, including the shortcut for when you just want your old site back.

What an Archive.org Download Actually Contains

Two storage worlds again. Item downloads give you finished files, a PDF is a PDF, an ISO is an ISO. Wayback-side downloads give you web capture data: HTTP responses frozen at crawl time, plus indexes describing them. The confusing formats all come from that second world, plus there is a single-file escape hatch that bridges both.

Sizes matter for planning. A single WARC segment from a crawl collection runs about 1 GB compressed. A mid-size site's CDX listing might be 50,000 lines of JSON. A single file is, well, one file. Match the format to the job before you start the transfer, not after, re-downloading a gigabyte because you grabbed the wrong thing is a rite of passage best skipped.

WARC: The Raw Capture Format

WARC, Web ARChive, ISO standard 28500, is a container that concatenates records. Each record is one captured HTTP exchange: the response headers followed by the exact payload bytes, plus metadata like capture time and target URL. Records come in types, response, resource, metadata, warcinfo, and files almost always arrive gzipped, one record per gzip member, so tools can seek without decompressing the whole archive.

• Nothing is rendered or cleaned. You get byte-for-byte what the crawler saw, including 404 pages and redirect chains.
• One WARC holds thousands of often unrelated URLs from a crawl, your site may be 2% of the file.
• You need tooling to read it: the Python warcio library for extraction, or ReplayWeb.page and pywb to replay it as browsable pages.

Extracting with warcio is a filter loop: open the archive, skip records whose target URL is outside your domain, write payloads to disk mirroring the original paths, dedupe identical digests. Fifty lines of Python, or one long evening of learning why people pay for this.

Where do you actually get WARCs? Two places. Archive Team rescue crawls and Internet Archive crawl collections ship them as ordinary item files, look for.warc.gz in the item's download box. And if you ever commission or export a crawl of your own, WARC is what comes back. It is the web archiving world's shipping container: ugly, standardized, and everywhere.

JSON CDX: The Index, Not the Content

CDX is the Wayback Machine's card catalog. Query web.archive.org/cdx/search/cdx?url=example.com&output=json and you get a JSON array where each row describes one capture: urlkey, timestamp, original URL, mimetype, statuscode, digest, and byte length. No page content at all, just a precise menu of what exists, capture by capture.

Two flags earn their keep on every query: filter=statuscode:200 drops the redirects and error pages, and collapse=digest removes duplicate captures of identical content. On one client's WooCommerce store, those two flags cut 38,000 raw rows down to 4,100 real, unique pages. That list became the restore plan, and the price estimate.

Query tricks worth knowing: append matchType=prefix to cover every URL under a path, and use from=2008&to=2012 to bound the years. A wildcard like url=example.com/* already implies a prefix match on the whole domain. Start narrow, an unfiltered domain query on a big site returns megabytes of JSON and a frozen browser tab.

• Scoping: how much of the site was ever captured, and in which years.
• Gap analysis: finding the image directories and CDN hosts that never got crawled.
• Batch fetch lists: feed the timestamp-plus-URL pairs to your downloader, one id_ request each.
• Estimates: Restorix reads this same CDX data to price a restore, file count, total size, locked price, upfront.

Single Files: The Easy Archive.org Download

The third format is barely a format: one original file, fetched straight. For items, that is the file link in the download box. For snapshots, it is the id_ trick, insert id_ after the timestamp, as in web.archive.org/web/20130501000000id_/http://example.com/logo.png, and the Wayback Machine returns the untouched payload with no toolbar and no rewritten markup.

Reach for single files when you already know exactly what is missing: the stylesheet that did not crawl, the PDF price list, the hero image a client keeps asking about. I keep a scratch folder of these per project. What you should not do is build a whole site this way, 4,000 manual id_ fetches is a script crying out to be written.

Two failure modes to expect. On the snapshot side, id_ only works if that exact URL was captured, check the CDX index first instead of guessing URLs. On the item side, the direct URL pattern archive.org/download/{identifier}/{filename} is stable and scriptable, but filenames with spaces need proper URL encoding or wget will 404 on a file that plainly exists.

Which Archive.org Download Format Do You Need?

The honest dependency chain for a website restore is CDX first, WARC or id_ fetches second, single files to patch holes last. Skipping the CDX step is how you download a gigabyte of crawl data and still miss half the site. Item downloads sit outside this chain entirely, they are finished goods, and if what you need is a book or a driver, stop reading and go click.

Working With What You Downloaded

Gotchas that bite everyone once. Gzipped WARCs are multi-member, a naive gunzip works, but streaming readers must not stop at the first member. Digests repeat across captures, so dedupe before you count files or your totals will lie. Character encodings from 2004 are not UTF-8; keep bytes as bytes until you must decode. And every Wayback-rewritten URL inside captured HTML still points at web.archive.org until you rewrite it back.

Plan the output structure before extracting: mirror the original URL paths, keep a manifest of which payload came from which record, and never overwrite a variant you might need later. Restorix can export exactly this kind of manifest (JSON or SQLite) with every restore, doing it by hand, you will understand why that feature exists.

Or Skip the Format Wrestling Entirely

Everything above is learnable, and none of it is a good use of a business owner's weekend. Restorix exists for exactly this layer: it reads the CDX data, pulls the captures, dedupes, rewrites links, and hands you a working site. The free estimate shows the archived file count, total size, and a locked price before you pay, first file restored free, pay per file after that, automatic refund to balance if anything fails.

Restore options map one-to-one to the pain points above: relative internal links instead of archive.org URLs, HTTPS conversion, stripping old analytics and ads, keeping 301 redirects intact. Deploy is one click to SSH/SFTP, FTP/FTPS, or S3 from the restore dashboard, and the bundled CMS at /webarchive-cms.php, randomly generated password, safe mode on by default, means you can edit the revived site without touching a line of WARC tooling. Formats are for archivists. You want your site back.