Webmasters can for instance filter out Google Bot from their website statistics to get a better picture of how many real users visit the site in a given time.

Some webmasters and services on the other hand try to cheat by allowing Google Bot access to all of their contents while they display a registration or buy page to users who want to access the same information.

That’s not allowed according to Google’s terms of use but some webmasters do it nevertheless.

Some users had now the idea to pose as Google Bot to access the information without buying or registering first.

Be The Bot is a website that simplifies the process. It contains a form where a web address can be entered. The user can also select to pose as Google Bot or Yahoo Bot. The requested url will then be displayed on the same screen.

Have you ever been googleing something, and you see exactly what you need in the preview, but when you click the link it doesnt show you what you want to see?
This is because the owners of the site are trying to trick you into buying something, or registering. It’s a common tactic on the internet. When Google visits the site, it gives something called a “Header”. This header tells the site who the visitor is. Google’s header is “Googlebot”. The programmers of the site check to see if the header says “Googlebot”, and if it does, it opens up all of its content for only googles eyes.

This works on all pages that allow Google Bot or Yahoo Bot complete access to their website but block visitors by asking them to register or buy first.

It works for instance on the Washington Post website which asks visitors to register before they can read the contents that are posted on the site. Copying the url from the website of the Post or opening washingtonpost.com in the url form at Be The Bot will provide unrestricted immediate access to the contents. (via Online Tech Tips)

This can be quite devastating especially for webmasters who do earn their living from their websites. It takes up to two weeks until changes can be seen when you edit the robots.txt file which is way to long if you make a mistake.

One great way to check if your robots.txt is valid and does exactly what you want it to do is to use check it in realtime at Google’s Webmaster Central service. The first thing that you have to do is to create a free account and add your site to it.

Once that is done you can access various services that are offered. One of them is the robots.txt analysis which lets you check your robots.txt on your website. Google automatically retrieves the robots.txt from your website if one exists and adds the main url to the list of urls that should be checked.

You may add new entries to the robots.txt and to the list of urls that should be checked. This is important because of two reasons. First, you want to check new entries or a complete new robots.txt file which means you have to add and edit entries.

It is also important to check various urls and not only the main url. If you take ghacks for example. All article pages have a certain syntax which differs from that of the main page. To give you an example, I did add the following robots.txt file and articles pages. This is the right way if you run a WordPress blog. If you do run a different website you do need to add a different robots.txt and pages of course..

robots.txt

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/

User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*?*
Disallow: /*?

# This is the ad bot for google
User-agent: Mediapartners-Google*

# Allow Everything
Allow: /*

Test URLs against this robots.txt file

http://www.ghacks.net/
http://www.ghacks.net/2007/05/20/support-ghacks/
http://www.ghacks.net/tag/
http://www.ghacks.net/category/
http://www.ghacks.net/2007/05/20/flitter-a-flickr-twitter-realtime-screensaver/trackback/

You may add a second search engine bot which should also try and crawl the site. It would be a good idea to select the Adsense bot for instance. Clicking on check displays the results if Google bot wanted to crawl your website.

Allowed means that Google Bot is able to crawl that type of sites will Blocked means that this type of sites will not be crawled. If the results are not to your satisfaction you can easily edit the robots.txt and check again until they are.

All you need to do is the following. Search for something, e.g. printing from a website, the search string you would enter would look like the following.

printing from a website site:experts-exchange.com

Please note that it is not necessary to use the site:experts-exchange.com parameter, this is only done for demonstration purposes.

The first search result will link directly to the experts-exchange.com article with no solution but a button stating View Solution. If you click that button you see that you will have to subscribe to view the solution. Not today though.

Go back to the Google results page and click on Cached which loads a cached version of that webpage. Guess what ? The cached page holds all the answers to the question right at the bottom. Just scroll down and see for yourself.