Google bot is the general term for Google’s automated web crawling service that is linked to the Google search engine. Google sends out requests to webpages that use a Google Bot user agent. This specific user agent is used for several purposes including identification and restrictions. Webmasters can for instance filter out Google Bot from [...]

Categories: The Web

A robots.txt tells the search engine robots about the directories and files that should be indexed on a website. A wrongly edited robots.txt file could mean that search engine robots are not crawling your website anymore with the result that new articles will not be indexed in the search engine index.

This can be quite devastating especially for webmasters who do earn their living from their websites. It takes up to two weeks until changes can be seen when you edit the robots.txt file which is way to long if you made a mistake.

Categories: Google, The Web

You might have stumbled upon this site if you ever had a computer related question that you could not answer. First step, ask google. Second step, find websites that force you to login or even pay to see the answer. Big thanks to eonestudio for making this little trick public. The above link displays a video which shows how you can take a look at the answers for free.

Categories: Hacking