Ever wondered why a Google search returned search results that lead to sites that require a registration ? How the hell did googlebot index the site without a registration ? Many sites want their site indexed in Google to receive more hits and thus, make more money with advertisement on their site. Even more interesting is the following question: How do we use this to our advantage ?
Its fairly easy, we have to disguise ourselves as the Google bot and voila, many sites let us in without registration. The parameter we have to change is called the User Agent. The user agent defines the browser and version that you use. For example it would show Internet Explorer as your browser if you are using this one.
You need to alter your settings to the following:
User Agent: Googlebot/2.1
Compatible: http://www.googlebot.com/bot.html
You can do so in Opera with ease. Firefox offers an extension which is downloadable from the official website.
For Internet Explorer you need to change registry entries.
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsCurrentVersionInternet Settings5.0User Agent]
@="Googlebot/2.1"
"Compatible"="+http://www.googlebot.com/bot.html"
Save this as ua.reg and execute.
To revert the changes back, you need the following:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsCurrentVersionInternet Settings5.0User Agent]
@="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
Save as oua.reg and execute.
Enjoy :)
Enjoyed the article?: Then sign-up for our free newsletter or RSS feed to kick off your day with the latest technology news and tips, or share the article with your friends and contacts on Facebook or Twitter.Related Articles:
Google Bot has privilegesGoogle Chrome Extension Reveals Website User Agent Detection
Access Websites As Google Bot
Check your robots.txt at Google
Megaupload bypass country limit
what have I to write in APP NAME, APP VERSION, I can’t find compatibility in EDIT new AGENT
How can you do it in Opera?It doesn’t allow other user agents except the standard 3 inside the browser and 5 if you use ua.ini
> You can do so in Opera with ease.
ummm, no you can’t. — at least ease would imply obvious, and it isn’t, even if it is possible at all…
Cunning…
You are of course right concerning opera. I was at work when I wrote this text and had no way to check on opera there.
I´am sorry for this misinformation.
Before you do this, be aware that many sites use bot filters (robots.txt) to prevent bots from cruising through parts of their website. You may get around registration, but not get to see any pictures.
I’ve installed the Firefox User Agent extension. I don’t find a “Compatible:” option. Also, what do I enter in the Description:, App Name:, App Version:, Platform:, Vendor: & Vendor Sub: options? Will be very grateful if I could be helped.
What have I to do in the case of Win98SE? This field is as follows:
REGEDIT4
[HKEY_USERS\.DEFAULT\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings]
“User Agent”=”Mozilla/4.0 (compatible; MSIE 6.0; Win32)”
I have read that some sites also check the IP to
see if the browser is really googlebot.