How to Find All Pages on a Website (Even Deleted Ones)

How many pages does our site have? It sounds like a question with a database behind it. It rarely is. Last year a client swore their site had about 400 pages. The sitemap said 1,900. A crawl found 3,400. The Wayback Machine's index listed 11,000 URLs the site had ever served, half of them long deleted. Four sources, four answers, and all four were correct about different things.

Here are the four methods I use to find all pages on a website, what each one actually covers, and how to merge them into one clean list you can act on.

Why it's hard to find all pages on a website

There is no master list unless the CMS happens to keep one, and even then it only knows about itself. Real sites accumulate pages that fall outside any single inventory: orphaned pages no menu links to, sections an old migration left behind, uploads referenced once in an email campaign, faceted URLs generated on the fly, entire subdomains someone set up in 2016 and forgot.

Each discovery method sees a different slice of that mess. The sitemap is the CMS's self-report. A crawler maps the link graph. The Wayback Machine's CDX index remembers history. Search Console reports what Google actually served to users. None is complete; together they get close.

Decide what the list is for before building it. A migration inventory, an SEO audit, and a dead-site recovery need the same raw material but different cleanup.

The sitemap: fastest way to find all pages on a website

Start at /sitemap.xml. If it 404s, check robots.txt for a Sitemap: directive, plenty of sites park the file at an odd path. WordPress core has served /wp-sitemap.xml since version 5.5; Yoast and Rank Math generate /sitemap_index.xml, which fans out into child sitemaps per post type. Fetch the index, then every child.

The protocol caps each sitemap file at 50,000 URLs and 50 MB uncompressed, so large sites always use indexes, do not stop at the first file you find.

Now the caveats, because sitemaps lie by omission. A sitemap only lists what the CMS knows about: no orphaned media uploads, no pages deleted years ago, no legacy sections running outside the CMS, nothing generated by that custom script in /old-tools/. And on hand-rolled or stale sitemaps, even the known list is fiction. Spot-check lastmod dates against pages you updated recently; if they disagree, trust nothing.

Crawl the site like Googlebot would

A crawler starts at the homepage and follows every link it can see, the same way a search engine does. Screaming Frog is the industry default and free up to 500 URLs; Sitebulb is friendlier for audits; for full control, a small Python script with requests and BeautifulSoup, or wget in spider mode, costs nothing.

Crawling finds what sitemaps miss: paginated archives nobody added to the sitemap, tag and category pages, broken links pointing at pages that still exist, redirect chains burning crawl budget.

Its blind spots are the mirror image of the sitemap's. If nothing links to a page, the crawler never finds it. JavaScript-rendered menus need a headless browser or the crawl stops at the homepage. And faceted navigation, filters, calendars, sort orders, can balloon a 500-page store into a 500,000-URL crawl trap; set exclusion rules before you press start, not after. On sites you do not own, keep concurrency low and honor robots.txt.

The Wayback CDX API: find all pages on a website, past and present

This is the method almost nobody uses and the one that finds the most. The Internet Archive's CDX server will list every URL it has ever captured for a host, including pages deleted a decade ago, which no live method can see.

One curl command gets the full inventory:

curl "https://web.archive.org/cdx/search/cdx?url=example.com/*&output=text&fl=timestamp,original,statuscode&filter=statuscode:200&collapse=urlkey&from=2010&to=2020"

The parameters that matter: collapse=urlkey dedupes repeat captures of the same URL; filter=statuscode:200 drops the 404s and redirects; matchType=domain widens the net to subdomains; from and to bound the years. For a dead site, that date window is where the gold is.

Expect volume. A long-lived forum or store can return millions of rows, and query-string noise, session IDs, tracking parameters, inflates the list. The API is free but rate-limited, so be patient with big domains and split huge pulls into yearly chunks.

If you want the answer without the plumbing, Restorix runs this same index for its free estimate, paste a domain and it reports the exact archived file count and total size in seconds, no queries required.

Google Search Console: find all pages your website shows Google

For sites you own, Search Console adds the one dataset nobody else has: what Google actually indexed and served. Verify ownership, then work two reports.

• Performance report, Pages tab: every URL that earned an impression, exportable. The web UI caps exports at 1,000 rows; the Search Analytics API pages far beyond that, and the bulk export to BigQuery delivers the full dataset daily.
• Indexing, Pages report: indexed versus crawled-currently-not-indexed, with sample URLs, a quick read on how much of the site Google even bothers to keep.

The unique value is the impression data: pages real users reached, including orphans nothing links to. If a forgotten landing page still pulls thirty clicks a month, it goes on the keep list. Bing Webmaster Tools offers the same reports if Bing traffic matters to you.

Merge, dedupe, and clean the list

Four sources, four formats, the merge is where the value appears. Dump everything into one CSV, then normalize before deduping, or the same page appears six times in six disguises.

Those disguises are not hypothetical. One real product page can show up as https://www.example.com/page/?utm_source=newsletter, http://example.com/page, example.com/page/#comments, example.com/page?fbclid=abc123, www.example.com/page/index.html, and EXAMPLE.com/page, six rows, one document. Case, protocol, www, parameters, fragments, and default filenames all have to collapse to a single canonical form before the dedupe pass means anything. Skip this and your 11,000-URL inventory is really 6,000 URLs wearing costumes.

• Lowercase the hostname; strip fragments (#section) entirely.
• Drop tracking parameters: utm_*, fbclid, gclid, ref. Sort the query parameters that remain.
• Pick one trailing-slash convention and enforce it.
• Collapse protocol and www variants to your canonical form.

After dedupe, classify each URL: live 200, redirecting, 404 now but archived, or gone entirely. That third bucket, deleted but present in the CDX data, is the one migrations forget and regret. A twenty-line Python script handles all of this for most sites; spreadsheets cope up to a few thousand URLs.

Quick comparison of the four sources:

You found every page, now what

For a migration, the list becomes your redirect map: every URL with impressions in Search Console or captures in the CDX data gets a 301 to its closest modern equivalent. For an SEO audit, the crawl and index columns expose thin sections and crawl traps. Both are a few afternoons of honest work.

If the list is for a dead site you want back, skip the hand-rebuild. Restorix’s restore service restores the whole thing from the Wayback Machine: the free estimate shows the exact file count and size with a locked price, you pay per restored file with the first one free, and cleaning options strip old analytics, convert links to relative, and minify assets. The result deploys to your hosting in one click, or exports as XML, CSV, or JSON with a full file manifest if you want the raw material instead. Either way, the enumeration work you just did tells you exactly what a good restore should contain.