Why Webmasters Should Analyze Their 404 Error Log
When you load a website in your browser, a lot happens in the background to render and display that website to you. Your computer makes a number of requests, one for each file that is somehow linked on the site. This can be images or multimedia contents for instance. Whenever a file cannot be retrieved because it does not exist, a 404 not found error is created for that file in the log. This may not affect the person loading the website, but it very well may provide the webmaster with information that something is not working correctly.
There is another case where 404 errors happen. When you enter a link into your browser or a program directly that is not existing. This may be an image that you want to load, a feed url or any other form of link that points to a non-existing element on the web server.
The primary reason for analyzing error logs is to resolve all, or at least the most popular causes for errors. If you notice that half of your users want to access a page that does not exist, you may want to redirect them to an existing page instead, and especially so if that page has been moved or is available under a different address.
Once you have created the redirect, users are automatically passed along to the new resource. It is barely visible for them.
Too many broken links and elements can also have a negative impact on a site's search engine rankings. It is likely that the quality score sinks when a certain threshold of broken links and pages is reached.
For WordPress, I like to use the Redirection plugin. It basically records all 404 not found errors in the interface, with the option to redirect those requests to other pages on the site.
All requests that returned a 404 not found error are listed on one of the log pages of the plugin. Here you see the date the requests took place, the requested page, the referrer if available, and the IP of the user or machine that made the request. A click on the plus icon at the end of the row adds the source url to the redirection form at the bottom of the page.
You then need to enter a target url - that is the url you want the visitors to be redirected to - to complete the process. While it is possible to use regular expressions or customize the matching and action filters, it is nothing that needs to be done to create the redirect.
But there are other reasons why you may want to check the error logs regularly. I'd like to share two examples.
I noticed lots of requests for apple-touch-icon-precomposed.png and apple-touch-icon.png in the logs that tried to load the images from the root directory of the site. I first thought it was a misconfiguration of the mobile theme and plugin, but found out later that Apple devices make those requests if the device owner adds the site to it. This is the equivalent of the favicon. The issue was easily resolved. I took the Ghacks logo, modified it to fit in a 100x100 png file, saved it under both names and uploaded it to the root directory of the server. After that, the errors were gone.
The second example that I'd like to share may have more serious consequences. I noticed that someone tested the site for known vulnerabilities. This was done by trying to load pages on the site that would trigger the vulnerability. The most recent attacker tried to use the Timthumb vulnerability by testing whether themes that made use of the plugin were installed on the server (they were not). Still, I knew someone was looking for vulnerabilities which made me monitor my site closer for the time being. A webmaster who had a theme with TimThumb installed on the other hand, might check out the developer site for an update to protect the site against the vulnerability.
Depending on the number of 404 you received per day, you may have little motivation to resolve the issues. If you get thousands, you may spend most of your day resolving those errors with little time for anything else besides that. But it gets better, at least that's what I'm telling myself. It also feels a bit like gold digging, redirecting invalid links to valid links. It first makes your visitors happy, as they do not see an error page but a - hopefully - related page, and it also can have a positive effect on the site's quality score and search rankings.Advertisement
Thanks for this article. Actually I’m doing the same investigations with redirection plugin these days :)
But i have a question that was born after i saw the screenshot with the missing png’s – is it ok to redirect visitors that searched or tried to reach the missing images to equivalent pages, not to images (if they not exist anymore)
Well that depends. If the image that is requested is embedded somewhere, it may not change a thing. If it is a direct request, it might bring some visitors to your site. I personally would not do it.
Martin, you could write the definitive bible on being a webmaster.
Something to think about should you ever think of writing a book.
Interesting idea, maybe a book about some of the finer aspects of being a webmaster, like the one above. A book of tips and tricks.
Exactly. “Webmaster Secrets To Mastering The Web” :)
Catchy title, I give it some thought ;)
Do think about it and go ahead a lock in that title now in case you want to use it later.
You have a wealth of material to draw on you have already written for gHacks
which supplies the advantage of a foundation for getting started.
Thats a pretty good plugin. I’ve seen people recommend ones which check for 404 errors, and then automatically redirect people. I think having the process to automated can be a downfall. Image renaming an important page, and then it automatically linking all the old links to the home page or something else. I think all webmasters should eyeball all the 404s.
I wrote a free tool call SiteCrawl – you can check your site for broken links, look at all the anchor text used and see what pages link to / from a page. Worth a look :) http://sitecrawl.net/
Thanks again for the article, was a good read.
Thanks for sharing. I’ve found your information very helpful.
First, thanks to your site I became aware of this plugin. Within seconds I see the logs flood with listing of images for 404s. Basically I have a pop-out script for all those search engines/sites that frame my pictures and when users click on the listings in google, bing, etc. it pops out but apparently creates a 404. Is this bad? I haven’t seen any bad effects and have had it in place for a while now.
I also have a question on the redirect via the plugins. I am used to doing the redirects in htaccess being sure what response code is returned. I always use 301. Which one is it here with the plugin?
In the above example, where you add your home page url to the plugin redirect, are you suggesting to redirect all the 404s to your homepage? If so, I disagree with that blanket rule for all. Some 404s need to stay 404s and not redirected.
Very good plugin. Helps a lot.
That redirect plugin i could never get it to redirect anything, it still shows 404’s.