Website downloader. How to choose the files limit?

Published: 2020-02-12

Our Website downloader system allows you to download up to 200 files from a website for free. If there are more files on the site and you need all of them, then you can pay for this service. Download cost depends on the number of files. How to find out how many files are really on the website and how much it will cost to download them?

First, you need to point out that the number of site files is almost always greater than the number of site pages . It will be the same only when all the pages of the site are pure html files, without pictures, CSS, scripts and so on. This can only be seen on the very first Internet site, which was created in 1991 - http://info.cern.ch/ . If your website does not look like an artifact from the early web, then it will have much more files. But how to count them?

Only the administrator can know exactly how many files are on the website. If you do not have full access to the website, then you can count the files only approximately. The easiest way to calculate how many files on it is to check what was indexed by Archive.org by using our recovering sites from The Web Archive system. Fill the "Domain" field, and left empty "To timestamp" and "From timestamp". Click the "Restore" button and wait for a screenshot of the site with file counting? it will come to your e-mail. It should know that this number shows only how many files were indexed by the Web Archive, not how many actually are on it now. They can be either more or less.

The next way is to calculate the number of pages in sitemap.xml. This file is usually located at yourwebsite.com/sitemap.xml or its position can be specified in robots.txt. From the obtained number of pages, you can roughly estimate how many files are on the site. On average, a website has 2 times more files than pages. But if the site contains a lot of graphics, then the files / pages ratio can be much higher.

If there is there is no Sitemap on the site, you can find out the number of pages in Google using a simple request https://www.google.com/search?q=site: yourwebsite.com. But it will show only the number of indexed pages, not how many pages are actually on the site.

Important notice! We do not recommend to download sites with automatically generated content or sites with automatically generated internal links. Such websites contain an "infinite" number of files.

The use of article materials is allowed only if the link to the source is posted: https://en.archivarix.com/blog/website-downloader/

Latest news:
2020.02.14
New Friday, new updates!
A lot of new and useful was done in Archivarix CMS:
- In Search and Replace, you can now filter by url date.
- Now external links from all pages of the site can be deleted with the click of a button. Anchors are preserved.
- The new ACMS_SAFE_MODE parameter, which prohibits changing the Loader / CMS settings and loading custom files, is also prohibited from importing import settings and custom files.
- The JSON settings files for the Loader and CMS can now be downloaded to your computer and downloaded to the CMS from a file on the computer. Thus, the transfer of settings to other sites has become even easier.
- Creating custom rules has become more convenient, there are often used patterns that you can choose.
- New custom files can be created in the file manager without having to download the file.
- The url tree for the main domain always comes first.
- If you hide the url tree for the domain / subdomain, then this setting is saved while working with the CMS.
- Instead of two buttons, open / collapse the url tree, now one that can do both.
- Creating a new URL was simplified and when creating, you can immediately specify the file from the computer.
- In the mobile layout, the main working part comes first.
- After each manipulation of the file, its size is updated in the database.
- Fixed buttons for selective history rollbacks.
- Fixed creating new urls for subdomains that contain numbers in the domain name.
2020.02.07
New portion of updates!
There is no need to change anything in the source code of the files.
- Now you can upload sites to the server by uploading to the server only one script from our Archivarix CMS.
- In order to change something in the CMS settings, you no longer need to open its source code. You can set a password or lower limits directly from the Settings section.
- To connect your counters, trackers, custom scripts, a separate "includes" folder is now used inside the .content.xxxxxx folder. You can also upload custom files directly through the new file manager in CMS. Adding counters and analytics to all pages of the site has also become convenient and understandable.
- Imports support a new file structure with settings and the "includes" folder.
- Added keyboard shortcuts for working in the code editor.

These and many other improvements in the new version. The loader has also been updated and works with the settings that the CMS creates.
2020.01.23
Another mega-update of Archivarix CMS!

Added very useful tools that allow the click of a button:
- clean all broken internal links,
- delete missing images,
- set rel = "nofollow" for all external links.

Now additional restores can be imported directly from the CMS itself. You can combine different restores into one working site.

For those who work with large sites or use poor hosting - all actions that previously could stop at the timeout of your hosting will now be divided into parts and automatically continue until they are completed. Want to make a replacement in the code of 500 thousand files? Import several gigabyte recovery? All this is now possible on any, even very cheap hosting. The timeout time (by default, 30 seconds) can be changed in the ACMS_TIMEOUT parameter.

Our loader (index.php) now works on both http and https protocols, regardless of the build parameters. You can force the protocol by changing the value of the ARCHIVARIX_PROTOCOL parameter.
2020.01.07
The next update of Archivarix CMS with the addition of new features. Now any old site can be correctly converted to UTF-8 with the click of a button. Search filtering has become even better, because results can be also filtered by MIME type.
2019.12.20
We have released the long-awaited Archivarix CMS update. In the new version, in addition to various improvements and optimizations, a very useful feature has been added for additional filtering of search results and full support for the tree structure of URLs for recoveries with a large number of files. For more details, see the Archivarix CMS script change log.