Why Webmasters Should Analyze Their 404 Error Log
When you load a website in your browser, a lot happens in the background to render and display that website to you. Your computer makes a number of requests, one for each file that is somehow linked on the site. This can be images or multimedia contents for instance. Whenever a file cannot be retrieved because it does not exist, a 404 not found error is created for that file in the log. This may not affect the person loading the website, but it very well may provide the webmaster with information that something is not working correctly.
There is another case where 404 errors happen. When you enter a link into your browser or a program directly that is not existing. This may be an image that you want to load, a feed url or any other form of link that points to a non-existing element on the web server.
The primary reason for analyzing error logs is to resolve all, or at least the most popular causes for errors. If you notice that half of your users want to access a page that does not exist, you may want to redirect them to an existing page instead, and especially so if that page has been moved or is available under a different address.
Once you have created the redirect, users are automatically passed along to the new resource. It is barely visible for them.
Too many broken links and elements can also have a negative impact on a site's search engine rankings. It is likely that the quality score sinks when a certain threshold of broken links and pages is reached.
For WordPress, I like to use the Redirection plugin. It basically records all 404 not found errors in the interface, with the option to redirect those requests to other pages on the site.
All requests that returned a 404 not found error are listed on one of the log pages of the plugin. Here you see the date the requests took place, the requested page, the referrer if available, and the IP of the user or machine that made the request. A click on the plus icon at the end of the row adds the source url to the redirection form at the bottom of the page.
You then need to enter a target url - that is the url you want the visitors to be redirected to - to complete the process. While it is possible to use regular expressions or customize the matching and action filters, it is nothing that needs to be done to create the redirect.
But there are other reasons why you may want to check the error logs regularly. I'd like to share two examples.
I noticed lots of requests for apple-touch-icon-precomposed.png and apple-touch-icon.png in the logs that tried to load the images from the root directory of the site. I first thought it was a misconfiguration of the mobile theme and plugin, but found out later that Apple devices make those requests if the device owner adds the site to it. This is the equivalent of the favicon. The issue was easily resolved. I took the Ghacks logo, modified it to fit in a 100x100 png file, saved it under both names and uploaded it to the root directory of the server. After that, the errors were gone.
The second example that I'd like to share may have more serious consequences. I noticed that someone tested the site for known vulnerabilities. This was done by trying to load pages on the site that would trigger the vulnerability. The most recent attacker tried to use the Timthumb vulnerability by testing whether themes that made use of the plugin were installed on the server (they were not). Still, I knew someone was looking for vulnerabilities which made me monitor my site closer for the time being. A webmaster who had a theme with TimThumb installed on the other hand, might check out the developer site for an update to protect the site against the vulnerability.
Depending on the number of 404 you received per day, you may have little motivation to resolve the issues. If you get thousands, you may spend most of your day resolving those errors with little time for anything else besides that. But it gets better, at least that's what I'm telling myself. It also feels a bit like gold digging, redirecting invalid links to valid links. It first makes your visitors happy, as they do not see an error page but a - hopefully - related page, and it also can have a positive effect on the site's quality score and search rankings.Advertisement