Search google (and update twitter) from the command line - gHacks Tech News

Search google (and update twitter) from the command line

There are some of us who prefer to do as much as possible from the command line. For whatever reason, this type of person wants to prove to the world the GUI is not necessary. For that reason, I thought I would illustrate how, with the help of curl, you can search Google and update your twitter status from the command line.

Curl is a tool that transfers data to or from a server. Curl supports HTTP, HTTPS, FTP, FTPS,  SCP,  SFTP,  TFTP,  DICT, TELNET,  LDAP  or  FILE protocols and works without user interaction. But the neat thing about curl is that it offers a ton of various tricks for the clever end user. One of those tricks is fetching and putting data to/from a server - like Google and Twitter.

What you'll need

In order to achieve these tasks you will need to have installed both curl and html2text. Since I am showing you how to use these tools for command line tricks, there's no need to figure up the GUI tools. So we will install both tools from the command line. To do so you need to open up your terminal and issue a command similar to:

sudo apt-get install curl html2text

You will be prompted for your sudo password and then asked to verify the installation. Hit 'Y' to continue on with the installation.

Once these two pieces of software are installed, it's time to work some "magic".

Searching Google

In order to do this you need to pipe one command through another. Piping is done with the help of the "|" character separating the commands. In this case you are going to pipe the curl command through the html2text command. What this will do is render the html fetched from Google into plain text.

In order to get curl to search a web site you will have to use a single argument with the command. The argument is:

-A

which instructs curl to send the user agent string to a web site so the site thinks it is actually a web browser doing the search. Crafty.

So the command looks like this:

curl -A Mozilla http://www.google.com/search?q='Jack Wallen' |html2text -width 80

Notice I use the single quotes. If you are searching for a single word this is not needed. If there is a space between two words, put them in single quotes. The results from this search would look like:

9. **** My Review of Jack Wallen's Post: �10 Reasons Why Linux Will ... ****
Windows 7 may be generating some positive buzz, but Jack Wallen remains
skeptical. In fact, he says it's only a matter of time before Linux takes
its ...
techxav.com/.../my-review-of-10-reasons-why-linux-will-triumph-over-
windows/ -Cached - Similar
10. **** Jack Wallen - news ****
Jack Wallen asks Why aren't schools adopting open source? and the answer
is simply exposure to open source. Getting thousands of students to call
and write ...
www.wikio.co.uk/news/Jack+Wallen -Cached - Similar

Because these results will fly by you might want to do one more trick with this command by sending it to a file instead. To do this we will use the ">" character and a file name like so:

curl -A Mozilla http://www.google.com/search?q='Jack Wallen' |html2text -width 80 > search

Once the search is done, issue the command less search and you will see the results of your search.

Update Twitter status

The same kind of trick is done with your twitter status. To do this you would issue a command like:

curl -u user:password -d status='Your tweet' http://twitter.com/statuses/update.xml

Where 'Your tweet' is the status update you want to enter.

Right after you issue that command a bunch of information will be returned. This information is your profile rendered in XML. I have yet to find a way to turn that off.

Final thoughts

The command line is an incredibly powerful tool. Imagine what you can do with these two examples. You could set up a cron job to update your status automatically. Or you could create a script to automatically gather data from Google searches. The possibilites are limitless.

We need your help

Advertising revenue is falling fast across the Internet, and independently-run sites like Ghacks are hit hardest by it. The advertising model in its current form is coming to an end, and we have to find other ways to continue operating this site.

We are committed to keeping our content free and independent, which means no paywalls, no sponsored posts, no annoying ad formats or subscription fees.

If you like our content, and would like to help, please consider making a contribution:

Comments

  1. k said on August 20, 2009 at 2:34 pm
    Reply

    you could always do -o /dev/null on the update command to stop the output from showing

  2. jack said on August 20, 2009 at 2:48 pm
    Reply

    @k: Oh yeah….’doh on my part. ;-) thanks for that.

  3. JohnDoe said on August 20, 2009 at 4:02 pm
    Reply

    curl -A firefox http://www.google.com/search?q='Windows 7′ | html2text -width 80
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 1350 100 1350 0 0 53143 0 –:–:– –:–:– –:–:– 1318k

    Google Error

    ****** Bad Request ******
    Your client has issued a malformed or illegal request.

  4. JohnDoe said on August 20, 2009 at 4:05 pm
    Reply

    whoops, forgot to add my question to that above
    ..
    if i do it w/o the html2text cmd, it spits the html out of course, html2text isn’t converting something properly? (yes it’s installed).

    Any idears?

  5. k said on August 20, 2009 at 4:41 pm
    Reply

    If you paste the output minus any user/password information something like pastebin I can have a look at it

  6. k said on August 20, 2009 at 7:10 pm
    Reply

    The reason is because your query string “?q=Windows 7”. For some reason it screws up google search. It must be some special keyword to do with numbers if you do “?q=Windows 1” the same thing occurs. You can get around it by doing “?q=Windows%207”, %20 is the hex value for space. You can use sed to convert lots of spaces something like:

    echo “http://www.google.com/search?q=Windows 7 is what I want” | sed ‘s/ /%20/g’

    then you could pipe that to curl using xargs by adding: | xargs -n1 curl -A mozilla

    Hope this helps

  7. JohnDoe said on August 20, 2009 at 7:18 pm
    Reply

    word. thanks k. makes plenty of sense now (duh!), didn’t even think of that

  8. k said on August 20, 2009 at 7:28 pm
    Reply

    It’s not obvious why “space[0-9]” causes a bug but I guess it has something to do with the commands google search runs eg. if you search for 1+1 it runs the calculator function. Must be something to do with that.

  9. metaeuphoria said on August 20, 2009 at 9:35 pm
    Reply

    I recently added surfraw to cygwin and in my linux installations. Works great with links terminal browser for a number of searches. Plus less typing.
    http://surfraw.alioth.debian.org/

    For twitter, I was able to get termtter to work in cygwin. http://termtter.org. I use it nearly everyday now. Didn’t get it to work in ubuntu for some reason; might be how I installed ruby. A ghacks tutorial would be great. :)

  10. Avinash said on August 21, 2009 at 6:45 am
    Reply

    Quitter a command line twitter client, can also be used for updating twitter from command line.

  11. aery said on August 21, 2009 at 1:47 pm
    Reply

    Goosh is another command line interface for Google search. So , ig you bassically love command line and cant leave windows, goosh should be the ideal choice for you.

  12. twitter said on August 21, 2009 at 2:20 pm
    Reply

    You should do a blog about how one can capture a. Fine as hell band dudes heart.

Leave a Reply

Check the box to consent to your data being stored in line with the guidelines set out in our privacy policy

Please note that your comment may not appear immediately after you post it.