How To Deal With Huge Apache Web Server Log Files

I must admit that I have not been paying attention to the size of some of the Apache web server log files on my dedicated server. For those that do not know, the Apache logs are the log files of the Apache web server. They log every hit to a website and provide excellent material that can be processed by a log file analysis software.

This log files can become huge if you have a popular website. The current size of the Ghacks log file is 45 Gigabytes. If it continues growing in that pace the web server will run out of hard disk space pretty soon.

The first problem was to figure out how to backup the 45 Gigabyte file on the web server since I did not want to loose all the data stored within. The idea was born to compress the Apache web server log file first and then reduce it to a few Kilobytes with the echo command.

The Apache web server log files are stored in the /var/log/apache2 directory on the Debian dedicated server. The first command to issue would compress the contents of the apache2 directory into multiple tar files if the compressed file reaches a size maximum. This makes it easier to transfer files to another computer.

tar -cz directory | split -d -b 1024m - destination.tgz

Directory is the source directory where the Apache web server log files reside and destination.tgz is the name of the to be created compressed file. It is important to note that you might need to stop the Apache web server for the operation to succeed because the compression utility will stop if files get changed during procession.

The last thing that is to be done is to clear the original Apache web server log files in the log files directory. This can be done with the following command even if Apache is running.

echo " " > /var/log/apache2/filename.log

This simply overwrites the contents of the file. Please note again that some Linux distributions use another directory structure and that the log files can be located in another directory on the server.
This takes only a few seconds and the log file will be cleared from all contents afterwards.

Then it is just a matter of transferring the various compressed archives to another computer.