The Firefox extension FireMirror uses a similar technique as the last suggestion to download websites to the local hard drive. It basically is a website mirroring software that can automatically download a website based on the user’s parameters. The default setting will for instance download the active page plus every page that is linked from that page. The depth, which is the parameter that defines the number of pages that the website downloader will download, can be configured in the program’s options. The maximum depth is 10 and the minimum 0.

Additional options include configuring the timeout duration, disabling link replacements, enabling reports or configuring filters to include urls with a specific string. The configuration can be saved as a profile to speed up future download runs.

A basic browser is provided in another tab that can be used to browse the website that is being downloaded but it is usually a better idea to load the downloaded pages instead from the hard drive (the browser will retrieve pages that have not been downloaded).

The local pages use relative paths so that the downloaded websites and pages can be moved around without breaking the navigation.

The extension is provided in an early version. There is for instance no stop or cancel button which means the process can only be stopped by clicking on the x button on the extension window. The report is also not working at this time.

FireMirror could become an interesting alternative to desktop software like HTTrack. The add-on is compatible with Firefox 3.6+ and can be downloaded from the Mozilla website.

Update: The Fire Mirror extension is no longer available. We suggest to use HTTrack which is linked above to download entire websites.

The last step in the configuration configures the capturing depths which can be set to first page, first page plus links or whole site. The same menu contains options to allow popups and to set the scanning to be careful which deactivates certain scripts and other potentially malicious contents. The download will start immediately after the last step. The program will display the download progress of all objects on the website.

Experienced users can define objects that should not be downloaded. Among them files like videos, cookies or scripts. Once the website or page has been downloaded it can be browsed in the program interface up to the level it was downloaded from the server.

The website downloader will display all pages that have been downloaded in the sidebar sorted by project name. A download will be initiated for every link pointing to a page that has not been downloaded before. The download speed depends on several factors including the connection speed of the computer system the application runs on.

Different view modes are available that differ from the default html view mode. It is possible to take a look at the contents, attached files (e.g. videos or images) or information (modification date, allowed objects, site and so on). Properties can be accessed for each downloaded website differently or combined for a project. They make it possible to set very specific rules for downloading contents including the number of page revisions to keep or the contents that should be downloaded.

Wysigot is not only a website downloader but also a website monitor. It can be set to monitor websites for changes and notify the user about those changes. Change verifications can be automatic, periodical or manual depending on the user’s choice. Alarms can be set to notify the user if a website has been updated.

The website monitor and downloader uses the rendering engine of Internet Explorer to display the website’s contents. It contains options to import Internet Explorer favorites which is convenient if several of those should be downloaded.

An alternative is the excellent Httrack which runs on Windows, Linux and OSX

All methods have various advantages and disadvantages. Using the Save As function in web browsers is probably the fastest way to download a page to the computer. The structure makes it on the other hand uncomfortable to work with on larger projects. Website downloaders on the other hand deal perfectly with large websites, they do require some knowledge and configuration though before they even start to download the first byte.

The online service Backupurl offers another way to create a cached copy of a website. The user enters the url of a page that he wants to preserve in the form on the website. The service will then cache that url for the user and provide two addresses to cached versions of the page. The main advantage of the service is that the cached pages are not stored locally. This might be favorable in environments with strict data storage policies. The disadvantage is obvious as well. Only one page can be cached per run which means it becomes as impracticable and uncomfortable as using Save As if multiple pages need to be cached. There is also no guarantee that the service will be there when the information need to be retrieved.

It would also be an interesting option to retrieve all pages that have been cached at once. The only way to keep track of all cached pages is to copy and paste all created urls into another document. Backup URL can be an interesting option under certain circumstance. Advanced users are better off with applications like HTTrack or similar applications.

One of the most efficient ways to rip websites is by using the program HTTrack which might look a little bit confusing at the beginning because of its many options. I would like to walk you through the process of ripping a website. Please note that this method is not working on all websites but on most.

To begin with you need to download and install the software HTTrack Website Copier. Start it once it has been installed, you will be greeted with a new project dialog. Each project creates the offline copy of one or more urls.

The first screen manages the properties of the project. Just add a name – i prefer the name of the website that I want to rip – and a location on your hard drive where you want to save it. Make sure you have enough free disk space on that hard drive. Click Next to continue.

You add urls and the kind of action that you want HTTrack to perform. The standard action will download an exact copy of the website and make it available offline. The most important aspect here is the Set Options button which opens the configuration for the project.

It is very important to load the options and make some changes there. Click on the Browser ID tab and change the ID to Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0). Some websites check for the default ID of Httrack and deny access to it. This way makes it possible to prevent that from happening.

Access the Limits tab afterwards. Select the maximum mirroring and external depths. The first defines how many links will be scanned beginning from the homepage. If you set that to 2 for instance the homepage will be scanned, page 1 which was linked from the homepage will be scanned and page 11 will be scanned which was linked from page 1.

If you leave the first option blank all links will be scanned on that website. No external links are scanned by default which can be changed in that menu as well. I suggest to leave it at that because it would really bloat the project. Make sure you increase the maximum transfer rate in the same menu to the maximum as well to ensure faster downloads.

The Scan Rules tab is another important one. You can include and exclude files in here. If you do not want to download .exe files for instance you can use the string “-*.exe” without the “” in the form.

Passworded Websites:

Passworded websites are most of the times harder to come by. You need to supply HTTrack with the username and password for that website. The easiest way to do so is to add it to the url in the main menu. Instead of adding the url http://www.example.com/ you would add it this way: http://username:password@www.example.com/

That’s for websites with basic authentication which means popups that ask for a username and password. It’s more difficulty if the website uses form based logins. Your best option to rip those websites is to click on the Add Url button in the main menu and use the capture url feature.

This requires you to set a proxy in your favorite browser for a short time and login into the website that you want to rip so that HTTrack can check the way it is done and hopefully emulate this way when ripping the website.

The question that some of you might be asking is why would someone want to rip a website ? You could rip websites for offline browsing purposes. Maybe you have a second computer without direct internet access but would like to view the website on this computer as well. A website that gives an overview over a programming language is a good example. Another reason would be to save the information that is on the website this way, maybe you fear that the website might be offline soon.

The HTTrack website offers a step by step guide that can be used to understand the main features of the tool and rip your first website using the settings of the tutorial. HTTrack is available for Windows and Unix, Linux & BSD.

WinHTTrack arranges the original site’s relative link-structure. Simply open a page of the ‘mirrored’ website in your browser, and you can browse the site from link to link, as if you were viewing it online. WinHTTrack can also update an existing mirrored site, and resume interrupted downloads. WinHTTrack is fully configurable, and has an integrated help system. NOTES: WinHTTrack is the Windows release of HTTrack.

HTTrack 3.44-1 Offline Browser Utility

HTTrack is a free offline browser utility that allows you to download www. Sites directly from the internet and arrange them in a local directory. This recursively builds all directories, HTML, images, and various other files from the server to your computer. HTTrack works with the original site and its relative link-structure.

Offline viewing is similar to collecting a library of websites, text and images. You are able to browse these sites as if you were online when you are actually offline. If you anticipate that you will be offline for any given reason and still need to access online information, this is a great way to do it. Of course, you can only view what you download to the local directory.

The result is the ability to browse any site from any link to any other link as long as they are downloaded to the local directory. All you have to do is open the mirrored page of the website in your browser as though you were viewing online. Offline viewing can be advantageous when you have multiple sites that you need to cross-reference while offline. Now HTTrack will update existing mirrored sites and any interrupted downloads will be resumed. It can be fully configured to needed specifications and includes an integrated help system to make it easy to use for beginners.

The Windows 200/XP/Vista/7 release of HTTrack is called WinHTTrack. The Linux/Unix/BSD release version is WebHTTrack. Most Unix versions are available, including Ubuntu, of course. This is convenient for users utilizing virtualization and running multiple operating systems. You can download as many versions of HTTrack as needed to fit VHDs or multiple boot systems. All versions are available on the download page, including versions for both 32-bit and 64-bit systems. Download from the following link:

http://www.httrack.com/page/2/

You will immediately notice that a version is available for virtually any operating system. Be sure to choose the right one or it simply will not work. This is only mentioned for those running more than one operating system on a single computer as mentioned above. It is an easy mistake to make; installing the wrong download to the wrong operating system.

Another link includes all Documentation for HTTrack. Anything that you need to know about the latest version of HTTrack is here. Simply click on the desired header for specifics. The information is too extensive to detail here, but you can find all of the specifics here:
http://www.httrack.com/html/index.html

For the truly technical users out there, you can get all of the specifics regarding the release changes from a link on the same page links above. For your convenience, the link to the release changes in the current edition is included here: http://www.httrack.com/history.txt

The utility is pretty much the same as before, with many minor changes that create improvements that may be significant to some and unapparent to other users. Regardless, this is a useful utility which any user can utilize as an offline browsing utility.