Web Articles: How to Extract Clean Text From Any Website

A client once called me in a panic: their agency had redesigned the company site and simply not migrated the blog. Forty-some posts, four years of search equity, gone. The agency had no backup. The first question out of their mouths was the right one: can we get the article text back?

That job, and the much smaller everyday job of saving one clean article from a cluttered page, are the same skill at different scales. Here is how I do both: which tools actually work, how to recover web articles from sites that no longer exist, and where the legal lines sit before you republish anything.

Why extraction is harder than it looks

On a typical news page, the article is under a fifth of the HTML. The rest is navigation, sidebars, cookie banners, newsletter popups, related-post carousels and tracking scripts. Copy-pasting from the browser drags all of it along, or worse, pastes text that reflows into chaos because it was built from nested divs and spans.

The usual obstacles:

• JavaScript rendering: the HTML you download is an empty shell until scripts run
• Pagination: one article split across five URLs for ad impressions
• Anti-bot walls: challenges that block simple downloaders on sight
• Lazy loading: images that only exist once you scroll
• Truncated RSS: feeds that stop after two sentences and an ellipsis

The method you pick depends on scale: one article, one whole site, or one dead site.

Method 1: reader mode for single web articles

For one page, do not build anything. Firefox's Reader View (F9, or the page icon in the address bar) strips a page to headline, byline and body text. Safari and Edge have their own versions, Edge will even read the article aloud. From reader mode, copy into your editor, or print to PDF for an archival-quality save.

1. Open the article and trigger Reader View
2. Fix the title and byline if the detector guessed wrong
3. Copy to your editor, or print to PDF
4. Save the original URL and capture date alongside the text, you will want provenance later

A lesser-known trick: many sites ship a print stylesheet. Hitting print preview sometimes gives you the whole article clean even when reader mode chokes, and 'Save as PDF' keeps the source URL in the header if you enable headers and footers in the print dialog.

The limits are real: one page at a time, metadata (author, date, tags) often lost or wrong, and it fails outright on paywalled, paginated or script-heavy pages. Pocket and Instapaper do the same cleanup with a save-for-later layer on top, handy for reading, useless for bulk.

Method 2: structured extraction of web articles

Past a dozen pages, hand work stops making sense. Structured extraction means running each page through a library that knows how to find the article inside the boilerplate, the same trick reader mode uses, but scriptable.

Before you write any extraction code, check two cheat codes. First, /wp-json/wp/v2/posts?per_page=100, if the site runs WordPress, that URL hands you every post as clean JSON, no scraping required. Second, sitemap.xml: it lists the site's URLs so you can enumerate articles instead of crawling blind. When you do crawl, keep it polite: one request per second, a real user-agent string, respect robots.txt, dedupe by canonical URL.

Extraction mistakes that waste hours

Four mistakes account for most of the cleanup work I see in extraction jobs, and all of them are cheaper to avoid than to fix:

• Trusting the detected date. Extractors happily grab the modified date, a comment date, or the year in the footer. Verify against the byline or the archive capture date.
• Losing the canonical URL. Save it with the text. Without it you cannot dedupe, set up redirects, or prove provenance later.
• Extracting list pages as articles. Category and tag pages look like content to a detector. Filter by URL pattern before you run the batch, not after.
• Ignoring encoding. A 2008 site in Windows-1251 becomes alphabet soup if you assume UTF-8, check the HTTP charset header, not just the meta tag.

Recovering web articles from a dead site

When the site is gone, the archive becomes your source. The Wayback Machine's CDX API lists every capture under a path, example.com/blog/*, and you can pull the latest good snapshot of each post, then run the same extraction tools against web.archive.org URLs. It works, but it is slow: the archive rate-limits, old captures reference missing images, and fifty articles turns into an afternoon of babysitting.

The faster route is restoring the site properly first. the website restore service gives you a free estimate, exact archived file count, total size, locked price, then restores the files over a date range of your choosing. The part most people miss: there is a structured article export option that hands you the article text as XML, CSV or JSON, plus a JSON or SQLite manifest of every restored file. The 2016 blog from my opening story came back that way, 412 files, 96 MB, posts exported to CSV with title, slug, date and body, re-imported into WordPress the same afternoon. The tutorial walks through the whole flow.

One more source people forget: Google retired its cache links in 2024, but archive.today often holds pages the Wayback Machine missed, especially news articles, and Bing still serves cached copies for some sites. Worth a lookup before you declare an article unrecoverable.

Copyright rules before you republish

Not legal advice, just the lines professionals actually work within. Facts and ideas are not copyrighted; the specific expression of them is. Your own articles are yours to do with as you please. For anyone else's text:

• Check for a license. Creative Commons marks matter: CC0 and CC-BY allow republication (CC-BY needs credit); CC-BY-NC bars commercial use; no mark means all rights reserved.
• Get permission in writing when there is no license. An email that says yes beats a page of fair-use theory.
• Fair use is a courtroom defense, not a permission slip. It weighs purpose, amount, nature and market effect, and quoting a whole article fails the amount test almost every time.
• Attribution and a canonical link are good manners and good SEO. They do not cure infringement.
• Images carry their own copyrights, separate from the text. Re-hosting them needs its own permission.

One nuance that bites agencies: work made by employees usually belongs to the company, but contractor work may not unless the contract says so. If you are recovering your own company's blog, confirm who actually owns the text before you republish it under a new brand.

A republishing checklist that keeps you safe

1. Confirm you own or have licensed every piece, text and images separately
2. Keep the original author name and publication date with the article
3. Set a canonical URL if the old version still exists anywhere
4. 301-redirect the old URLs to the new locations
5. Update internal links so they point at live pages, not dead ones
6. Note material edits ('updated July 2026') instead of silently rewriting history

Do these six things and a recovered blog picks up its search rankings surprisingly fast, Google treats a faithful restoration at the original URLs as the same site coming back, not a new one starting over.