Comments on: Google Bot has privileges http://www.ghacks.net/2007/08/15/google-bot-has-privileges/ A technology blog covering software, mobile phones, gadgets, security, the Internet and other relevant areas. Tue, 24 Nov 2009 21:14:15 -0600 http://wordpress.org/?v=2.8.6 hourly 1 By: simpleminded http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-429156 simpleminded Thu, 31 Jul 2008 10:53:54 +0000 http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-429156 In case that wasn't very clear, I'll simplify it. Websites don't grant Googlebot special access to their pages so that people go to them and have to register to see. <b>Googlebot just breaks in. It's not a lure to make you register.</b> In case that wasn’t very clear, I’ll simplify it.

Websites don’t grant Googlebot special access to their pages so that people go to them and have to register to see.

Googlebot just breaks in. It’s not a lure to make you register.

]]> By: simpleminded http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-429139 simpleminded Thu, 31 Jul 2008 10:30:38 +0000 http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-429139 Actually it never was "a trick". It never worked to begin with. It's just been people saying "oh I bet this would work". It doesn't work because the concept is flawed. Googlebot does indeed have special access to "registered only" pages, but it's not because of its user-agent, IP address, or anything like that. Websites (forums particularly) don't allow Googlebot special permissions, and they don't check to see if Googlebot's IP or user-agent is spoofed, unless an admin does it manually out of suspicion. The most they do is disallow certain parts of the site to Googlebot through the use of Robots.txt. The REAL reason Googlebot gets into "registered only" pages is because the website didn't disallow Googlebot from crawling them in Robots.txt, and Googlebot just breaks in essentially, because the only rules it follows are those defined in Robots. I'm not sure what methods it uses to do such, but as an admin on several forums, I know for fact that it just "breaks into" our registered users only pages, as I’ve seen the Googlebot show up in logs as accessing anything from the main index to user profiles and private messages, even the Moderators Only sections of the forum, as a guest (or anonymous) user. The only way to stop it is Robots.txt or to ban the entire Googlebot IP range (and even that doesn't work for long because there's always new IP ranges). So pretending you're Googlebot will never, ever work. You have to actually BE the Googlebot software. Actually it never was “a trick”. It never worked to begin with. It’s just been people saying “oh I bet this would work”. It doesn’t work because the concept is flawed. Googlebot does indeed have special access to “registered only” pages, but it’s not because of its user-agent, IP address, or anything like that.

Websites (forums particularly) don’t allow Googlebot special permissions, and they don’t check to see if Googlebot’s IP or user-agent is spoofed, unless an admin does it manually out of suspicion. The most they do is disallow certain parts of the site to Googlebot through the use of Robots.txt.

The REAL reason Googlebot gets into “registered only” pages is because the website didn’t disallow Googlebot from crawling them in Robots.txt, and Googlebot just breaks in essentially, because the only rules it follows are those defined in Robots.

I’m not sure what methods it uses to do such, but as an admin on several forums, I know for fact that it just “breaks into” our registered users only pages, as I’ve seen the Googlebot show up in logs as accessing anything from the main index to user profiles and private messages, even the Moderators Only sections of the forum, as a guest (or anonymous) user.

The only way to stop it is Robots.txt or to ban the entire Googlebot IP range (and even that doesn’t work for long because there’s always new IP ranges).

So pretending you’re Googlebot will never, ever work. You have to actually BE the Googlebot software.

]]> By: Nobody http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-195922 Nobody Sun, 11 Nov 2007 13:28:42 +0000 http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-195922 Just use Google's cache of the page. Just use Google’s cache of the page.

]]> By: John http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-163911 John Sat, 18 Aug 2007 04:25:03 +0000 http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-163911 hey anyone know how to do so in opera browser? like i do opera:config then go to user agent but i dont know how to add google bot. hey anyone know how to do so in opera browser? like i do opera:config then go to user agent but i dont know how to add google bot.

]]> By: Threshold http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-163836 Threshold Fri, 17 Aug 2007 00:55:46 +0000 http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-163836 Thanks Martin, I tried several different combinations of googlebot found on the Internet but it hasn't worked for me on any website I have tried so far. I guess webmasters have wised up since this trick has been around for almost 3 years. Anybody has had any more luck with this? Thanks Martin,
I tried several different combinations of googlebot found on the Internet but it hasn’t worked for me on any website I have tried so far.

I guess webmasters have wised up since this trick has been around for almost 3 years.

Anybody has had any more luck with this?

]]> By: Martin http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-163777 Martin Thu, 16 Aug 2007 06:11:29 +0000 http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-163777 Threshold try the following settings: Description: Googlebot 2.1 User-Agent: Googlebot 2.1 Threshold try the following settings:
Description: Googlebot 2.1
User-Agent: Googlebot 2.1

]]> By: Threshold http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-163768 Threshold Thu, 16 Aug 2007 02:14:12 +0000 http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-163768 Well, in case this actually works, it would be nice if you explained how to fill the Googlebot details in User-Agent-Switcher for the teckie-challenged out here. What goes into each field? This trick is reported all over the net but not one that explains it clearly. Well, in case this actually works, it would be nice if you explained how to fill the Googlebot details in User-Agent-Switcher for the teckie-challenged out here.

What goes into each field?

This trick is reported all over the net but not one that explains it clearly.

]]> By: ser0 http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-163731 ser0 Wed, 15 Aug 2007 09:42:25 +0000 http://www.ghacks.net/2007/08/15/google-bot-has-privileges/#comment-163731 I think everyone should report sites like EE to Google for violating their policy of showing the bot one page and giving users a completely different page. I only used the cached version of EE pages now as the normal linked page is completely useless. I think everyone should report sites like EE to Google for violating their policy of showing the bot one page and giving users a completely different page.

I only used the cached version of EE pages now as the normal linked page is completely useless.

]]>