Disclaimer: You should carefully consult the terms of service of the Internet Archive and the TOS and copyright of any site you try this on. I make no promises as to the legality of this for any particular website.
This article is intended for people pretty comfortable with the command line. Should you need help, please email me and I will do what I can to assist.
Data on the internet is often ephemeral. This is what it is. Often you can't find what you were looking for because it's been moved or deleted. Most of the time people should be able to be forgotten, but sometimes there's important reasons why we need to try and save information that people have deleted.
The Internet Archive and the Wayback Machine are invaluable resources that work to save our digital history. By searching the wayback machine you can find all kinds of information that may have been lost for any number of reasons, like hosting lapsing or company existence failure. But you still can't download an entire site. I found a pretty neat tool, the Wayback Machine Downloader, that allows for the fetching of an entire site, so even if the site disappears from the Internet Archive, we will still have a record of it.
To install the downloader, we can just use
gem install, or the script can be
$ git clone https://github.com/hartator/wayback-machine-downloader $ cd wayback-machine-downloader $ ruby bin/wayback_machine_downloader Usage: wayback_machine_downloader http://example.com Download an entire website from the Wayback Machine. Optional options: -d, --directory PATH Directory to save the downloaded files into Default is ./websites/ plus the domain name -s, --all-timestamps Download all snapshots/timestamps for a given website -f, --from TIMESTAMP Only files on or after timestamp supplied (ie. 20060716231334) -t, --to TIMESTAMP Only files on or before timestamp supplied (ie. 20100916231334) -e, --exact-url Download only the url provied and not the full site -o, --only ONLY_FILTER Restrict downloading to urls that match this filter (use // notation for the filter to be treated as a regex) -x, --exclude EXCLUDE_FILTER Skip downloading of urls that match this filter (use // notation for the filter to be treated as a regex) -a, --all Expand downloading to error files (40x and 50x) and redirections (30x) -c, --concurrency NUMBER Number of multiple files to dowload at a time Default is one file at a time (ie. 20) -p, --maximum-snapshot NUMBER Maximum snapshot pages to consider (Default is 100) Count an average of 150,000 snapshots per page -l, --list Only list file urls in a JSON format with the archived timestamps, won't download anything -v, --version Display version
Then it is simple enough to run it using
wayback_machine_downloader [site], or
if a site has been deleted, we can use the
-t option to fetch an older version
of the site. For example
-t 20200601 to fetch the latest version up to June
first of 2020.
Sites can be very very large, they can be compressed using any standard tool
like tar and insert compression utility here, or zip, or 7zip, the list goes
on. Another way to deal with this is to filter the HTML into other forms like
text. Utilities that can do this include the
html2text utility. Or
this little script
that's part of the aerc email program which I also replicate below.
#!/bin/sh # aerc filter which runs w3m using socksify (from the dante package) to prevent # any phoning home by rendered emails export SOCKS_SERVER="127.0.0.1:1" exec socksify w3m \ -T text/html \ -cols $(tput cols) \ -dump \ -o display_image=false \ -o display_link_number=true
I hope this helps us preserve our collective memory of the online spaces we share better so we can not forget what is important to us, even if information may try to hide from us.