Linux command line tips: wget
One of the applications I use most often is the wget tool. You don't realize how useful this tool is until you've used and then needed it but not had it available. Wget is, for all intent and purpose, a non-interactive web downloader that allows the user to, from the command line, download files without requiring the user to even be logged in (hence the "non-interactive" in the title.)
Wget can do a lot of things browsers can not do, which makes it an incredibly valuable tool for users who do a lot of downloading. Wget can even re-connect and finish a download if a connection is dropped. Let's take a look and see how to make best use of this very powerful tool.
Installation for wget is very simple, as it will be found in the standard repositories. You can install wget from either the command line (such as yum install wget or sudo apt-get install wget) or you can also install wget from the Add/Remove Applications tool (Ubuntu Software Center, PackageKit, Synaptic, etc) by doing the following:
- Open the tool.
- Search for "wget" (no quotes).
- Mark for installation.
Remember, wget is a command line only tool (although there are front-ends available) so you will need to either be working from within a terminal window or logged onto a virtual terminal. Now that you have the right tool open, let's see how wget can help you.
The basic usage is:
Where ADDRESS_TO_FILE is the actual address (URL or IP) to the file you want to download.
The above will download a single file to your hard drive. Now, what if you have a need to do an recursive downloading? For example there is an entire directory of files you need to download. To do recursive downloading on a remote directory the command would be:
wget -r ADDRESS_TO_DIRECTORY/DIRECTORY
The above command will download a directoryÂ hierarchyÂ that matches the hierarchy downloaded from. In other words, within the directory you ran the wget command, you will find (using the above as an example) a newly created directory called ADDRESS_TO_DIRECTORY. Within that directory will be all of the subdirectories you downloaded.
If you don't want to download the hierarchy, just the files, you would add the -nd switch like so:
wget -r -nd ADDRESS_TO_DIRECTORY/DIRECTORY
Now the only thing that will download is the files contained within DIRECTORY.
What if the address or directory you are downloading requires a username and password? You can pass that along with the wget command like so:
wget -r -nd --user=USERNAME --password=PASSWORD ADDRESS_TO_DIRECTORY/DIRECTORY
Where USERNAME is the required username, PASSWORD is the required password, ADDRESS_TO_DIRECTORY is the address and DIRECTORY is the directory containing the files.
Now, if you want to run wget in the background (so you don't have to be logged in) you would issue the command as such:
wget -bqc ADDRESS_TO_FILE/FILE
Where ADDRESS_TO_FILE is the address containing the file and FILE is the file name to be downloaded.
You will find wget to be one of the more useful tools in your Linux toolkit. And once you start using it you will never look back.