How to restore websites from the Web Archive - archive.org. Part 1

Published: 2019-12-04

Web Archive Interface: Instructions for the Summary, Explore, and Site map tools.

In this article we will talk about the Wayback Machine itself and how it works.

For reference: Archive.org (Wayback Machine - Internet Archive) was created by Brewster Cale in 1996 about at the same time when he founded Alexa Internet, a company that collects statistics on website traffic. In October of that year, company started archiving and storing copies of web pages. But the current form called WAYBACKMACHINE, that we can use now, started only in 2001, although the data has been saved since 1996. Web archive advantage for any website is that it saves not only the html-code of the pages, but also other file types: doc, zip, avi, jpg, pdf, css. A complex of html-codes of all page elements allows you to restore the site in its original form (on a specific indexing date when web archive spider visited the site’s pages).

So, archive is located at http://web.archive.org/. Let’s touch on web archive possibilities on the example of a well-known website as WhatsApp.

Enter website domain on the main page in the search field. In our case it will be whatsapp.com.

After entering website link, we see the calendar of saving html code of the page. On this calendar, we see notes in different colors on the save dates:

Blues means server valid code 200 response (no errors from the server);

Red (may be yellow or orange, depending on the browser and PC operating system) means 404 or 403 error, something that is not interesting when restoring;

Green stands for pages redirect (301 and 302).

Calendar colors do not give a 100% guarantee of compliance: on the blue date there can a redirect as well (not at the header level, but, for example, in the html code of the page itself: in the refresh meta tags (screen refresh tags) or in JavaScript).

Let’s go to 2009, when this website indexing (saving) started.

We see the version dated January 24, and then we open it in a new tab (in case of errors during operation, it is better to open the web archive tool in incognito mode or in another browser).

So, we see the 2009 version of the WhatsApp page. In the page url we see the numbers called timestamp, i.e. year, month, day, hour, minute, second, when this url was saved. Timestamp format (YYYYMMDDhhmmss).

Timestamp is not the time website copy saving, not the time of page saving, but this is the time of specific file saving. This is important to know when restoring content from a web archive. All website elements as pictures, styles, scripts, html code and so on have their own timestamp, that is, archiving date. 

In order to return from website page to the calendar, click on the link with captures number (page captures).

Summary tool

Select Summary tool on web archive main page. These are graphs and charts for website saving. All graphs and tables can be viewed by year.

The most useful information on this page is New URLs column sum. This amount shows us the number of unique files contained in web archive.

Due to the reason that web archive itself could index the page with or without www, this number may be approximate. It means that the same page and its elements can be located at different addresses.

Explore tool

It uploads into the table all the url that were previously indexed by the web archive spider.

Here you can find:

  1. MIME element type;
  2. Initial date of element indexing;
  3. The last date when element was saved;
  4. Total number of element captures (savings);
  5. Number of duplicates;
  6. Number of unique content savings by url.

You can specify any part of the element you are looking for in a filter field: in order to search for website content that is difficult to detect among large number of links.

You can enter part of the path, for example, the path to the folder (required with an asterisk), you can see all the url for the given path (all files from the page or folder) to analyze the indexing of this content.

Site Map tool

Click on the appropriate Site Map link on the website main page.

This is a doughnut chart with a separation by year to analyze the elements saved by web archive (which pages) in the context from the main url to the url of the second and n-th level. This tool allows you to determine the year when web archive stopped saving website new content or certain url copies (any response code, except for 200 code).

In the middle of the main page and then along the path structure in the second or third stage, we see website internal pages. There are no other kinds of files, only url saved. It means that we can understand where the archive was able or not to index pages.

Diagram shows:

  1. Main page

  2 - 5. Website pages nesting levels

In addition, by using this tool we can see the internal pages by structure and open them separately in a new tab.

Thus, having selected links to pages and elements with the required date of saving in web archive and having built the structure we need, we can proceed to the next step ― preparing the domain for restoring. But we will talk about this in the next part.

 

How to restore websites from the Web Archive - archive.org. Part 2

How to restore websites from the Web Archive - archive.org. Part 3

The use of article materials is allowed only if the link to the source is posted: https://en.archivarix.com/blog/1-how-does-it-works-archiveorg/

Latest news:
2020.02.14
New Friday, new updates!
A lot of new and useful was done in Archivarix CMS:
- In Search and Replace, you can now filter by url date.
- Now external links from all pages of the site can be deleted with the click of a button. Anchors are preserved.
- The new ACMS_SAFE_MODE parameter, which prohibits changing the Loader / CMS settings and loading custom files, is also prohibited from importing import settings and custom files.
- The JSON settings files for the Loader and CMS can now be downloaded to your computer and downloaded to the CMS from a file on the computer. Thus, the transfer of settings to other sites has become even easier.
- Creating custom rules has become more convenient, there are often used patterns that you can choose.
- New custom files can be created in the file manager without having to download the file.
- The url tree for the main domain always comes first.
- If you hide the url tree for the domain / subdomain, then this setting is saved while working with the CMS.
- Instead of two buttons, open / collapse the url tree, now one that can do both.
- Creating a new URL was simplified and when creating, you can immediately specify the file from the computer.
- In the mobile layout, the main working part comes first.
- After each manipulation of the file, its size is updated in the database.
- Fixed buttons for selective history rollbacks.
- Fixed creating new urls for subdomains that contain numbers in the domain name.
2020.02.07
New portion of updates!
There is no need to change anything in the source code of the files.
- Now you can upload sites to the server by uploading to the server only one script from our Archivarix CMS.
- In order to change something in the CMS settings, you no longer need to open its source code. You can set a password or lower limits directly from the Settings section.
- To connect your counters, trackers, custom scripts, a separate "includes" folder is now used inside the .content.xxxxxx folder. You can also upload custom files directly through the new file manager in CMS. Adding counters and analytics to all pages of the site has also become convenient and understandable.
- Imports support a new file structure with settings and the "includes" folder.
- Added keyboard shortcuts for working in the code editor.

These and many other improvements in the new version. The loader has also been updated and works with the settings that the CMS creates.
2020.01.23
Another mega-update of Archivarix CMS!

Added very useful tools that allow the click of a button:
- clean all broken internal links,
- delete missing images,
- set rel = "nofollow" for all external links.

Now additional restores can be imported directly from the CMS itself. You can combine different restores into one working site.

For those who work with large sites or use poor hosting - all actions that previously could stop at the timeout of your hosting will now be divided into parts and automatically continue until they are completed. Want to make a replacement in the code of 500 thousand files? Import several gigabyte recovery? All this is now possible on any, even very cheap hosting. The timeout time (by default, 30 seconds) can be changed in the ACMS_TIMEOUT parameter.

Our loader (index.php) now works on both http and https protocols, regardless of the build parameters. You can force the protocol by changing the value of the ARCHIVARIX_PROTOCOL parameter.
2020.01.07
The next update of Archivarix CMS with the addition of new features. Now any old site can be correctly converted to UTF-8 with the click of a button. Search filtering has become even better, because results can be also filtered by MIME type.
2019.12.20
We have released the long-awaited Archivarix CMS update. In the new version, in addition to various improvements and optimizations, a very useful feature has been added for additional filtering of search results and full support for the tree structure of URLs for recoveries with a large number of files. For more details, see the Archivarix CMS script change log.