Download an Entire Website from Archive.org, Not Just the Homepage

Anyone can save a homepage. The homepage is the easy part, it is the 4,000 files behind it that separate 'I grabbed a copy' from 'I actually have the site'. When someone asks me to download an entire website from archive.org, the homepage is usually already sitting on their desktop, and the stylesheet is not.

This piece is about the word 'entire'. What a complete copy actually includes, why the archive scatters your assets across years of different timestamps, and a checklist you can run at the end to prove you got everything instead of hoping you did.

What 'entire' means when you download an entire website from archive.org

A complete copy has four layers, and most people stop after the first. The HTML for every page that existed. The assets every page references: images, stylesheets, JavaScript, fonts, PDFs, videos. Internal links rewritten so the copy browses on its own, without leaning on archive.org's servers. And a manifest of what was captured, so the gaps are documented rather than discovered later by a client.

• Pages: every URL the crawler ever saw, not just the ones in the main menu. Old sites hide whole sections behind forgotten footer links.
• Assets: images, CSS, JS, fonts and downloadable files, on most sites these outnumber the pages five to one.
• Links: rewritten to relative paths, or your copy only works for as long as archive.org does.
• A manifest: a file-by-file list of what you have, turning 'I think I got it all' into something checkable.

The file counts surprise people. A modest 50-page business site routinely totals 400-800 files. A 2009 phpBB forum with user avatars and attachments can run into the tens of thousands. Plan for the file count, not the page count.

There is also a difference between a copy that browses and a copy that redeploys. A browsing copy just needs the files and working links. A copy you intend to put back on hosting wants more: consistent canonical URLs (pick www or non-www), HTTPS-ready references, and dead weight like decade-old analytics scripts stripped out. Decide which of the two you are building before you start, because retrofitting either is tedious.

Why assets live under different timestamps

The Wayback Machine does not photograph a whole site in one afternoon. Its crawlers fetch whatever they can reach, whenever they happen to reach it. Your 2014 homepage might reference a logo last captured in 2012, a stylesheet captured six times across five years, and a hero image captured never. When you view that page, archive.org quietly substitutes the closest capture it holds for each asset. That is why the page looks fine in your browser even though its parts are scattered across a decade.

Two consequences follow. First, there is no single 'timestamp of the site' you can point a downloader at, a complete download pulls each file from its own best capture. Second, the closest-capture trick hides gaps: wayback will happily serve a 2016 image into your 2014 page without mentioning it. Harmless for casual browsing, worth knowing about when you care whether the restored page is really the 2014 one.

An archived page is a collage. The HTML, the images and the CSS were each frozen at different moments, and the archive glues them together again every time someone looks.

Any tool or script you use needs a strategy for this. The lazy strategy, fetch everything at one timestamp, is exactly how you end up with a page full of broken image icons, because half the assets simply have no capture that week. The right strategy is per-file: take each asset from its own closest good capture within your date range.

How to download an entire website from archive.org: start with the inventory

Before fetching a single file, list what the archive actually holds. The CDX API gives you the full ledger for a domain: every captured URL with its timestamps, MIME type and HTTP status. Filter to statuscode 200, collapse duplicate content digests, and you have a realistic shopping list instead of a guess.

Read the inventory before downloading and you learn the uncomfortable things early. On that phpBB forum job, the CDX listing showed 11,000 captured URLs, and about 3,000 of them were the same avatar GIF saved under different query strings. Collapsing duplicates turned a frightening job into an ordinary one. The same listing also exposes the pages that were never captured at all, which no download method on earth can conjure back. Better to know that on day one.

Group the survivors by MIME type and the shape of the job appears: how many HTML pages, how many images, how much JavaScript, whether there are PDFs or video to worry about. That grouping drives both your download order and your final completeness check.

Checklist to download an entire website from archive.org without gaps

1. Pull the CDX inventory for the domain, filtered to HTTP 200 captures, covering your full date range.
2. Collapse duplicate content digests so identical files download once instead of hundreds of times.
3. Group the list by MIME type: HTML pages first, then CSS, JS, images, fonts, documents, media.
4. Download each file from its own best capture, using the id_ modifier to get original bytes without wayback's injected markup.
5. Rewrite internal links to relative paths so the copy works on any host, including your laptop.
6. Serve the folder locally and click through it. Broken links, missing images and font 404s show up within minutes.
7. Compare the final file count against the inventory and write down what is missing and why.

That last step is the one everyone skips, and it is the difference between a backup and a pile of files. A manifest turns 'I think I got it all' into a list you can audit. Restorix bakes this in: the free estimate reports the exact archived file count and total size before you start, and the restore can export a JSON or SQLite manifest of every file it brought back, so completeness is a number, not a feeling.

The gaps you will hit anyway

• Pages the crawler never found. If nothing on the public web linked to a page, wayback probably never saw it.
• Robots.txt exclusions. Older crawls skipped disallowed paths entirely, and some sites blocked archiving on purpose.
• JavaScript-rendered content. Crawlers historically stored the HTML shell, not whatever the scripts built afterward.
• Logins, carts and search results. Anything behind a POST request or a session is a wall the crawler could not pass.
• URL noise. Session IDs and tracking parameters inflate the inventory with thousands of duplicate 'pages' that dedupe down to a few real ones.

None of these are failures of your method, they are holes in the archive itself. The value of the inventory is that it shows you the holes before you promise anyone a complete restore.

When you do find holes, document them. A one-page note listing the twelve pages that were never captured, the video that only exists as a thumbnail, and the forum section blocked by robots.txt turns an awkward surprise into a managed expectation. Clients forgive missing files. They do not forgive discovering them on their own.

What a complete copy is worth

If the site is a client's store or a community forum with a decade of posts, a half-download is a liability dressed up as a backup. Either budget the hours to run the checklist yourself, or hand the crawling, timestamp matching, link rewriting and manifesting to a tool built for exactly this job. The estimate costs nothing, and the alternative is finding the missing 300 images after the site goes live.