Search google (and update twitter) from the command line
There are some of us who prefer to do as much as possible from the command line. For whatever reason, this type of person wants to prove to the world the GUI is not necessary. For that reason, I thought I would illustrate how, with the help of curl, you can search Google and update your twitter status from the command line.
Curl is a tool that transfers data to or from a server. Curl supports HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE protocols and works without user interaction. But the neat thing about curl is that it offers a ton of various tricks for the clever end user. One of those tricks is fetching and putting data to/from a server - like Google and Twitter.
What you'll need
In order to achieve these tasks you will need to have installed both curl and html2text. Since I am showing you how to use these tools for command line tricks, there's no need to figure up the GUI tools. So we will install both tools from the command line. To do so you need to open up your terminal and issue a command similar to:
sudo apt-get install curl html2text
You will be prompted for your sudo password and then asked to verify the installation. Hit 'Y' to continue on with the installation.
Once these two pieces of software are installed, it's time to work some "magic".
Searching Google
In order to do this you need to pipe one command through another. Piping is done with the help of the "|" character separating the commands. In this case you are going to pipe the curl command through the html2text command. What this will do is render the html fetched from Google into plain text.
In order to get curl to search a web site you will have to use a single argument with the command. The argument is:
-A
which instructs curl to send the user agent string to a web site so the site thinks it is actually a web browser doing the search. Crafty.
So the command looks like this:
curl -A Mozilla http://www.google.com/search?q='Jack Wallen' |html2text -width 80
Notice I use the single quotes. If you are searching for a single word this is not needed. If there is a space between two words, put them in single quotes. The results from this search would look like:
9. **** My Review of Jack Wallen's Post: �10 Reasons Why Linux Will ... ****
Windows 7 may be generating some positive buzz, but Jack Wallen remains
skeptical. In fact, he says it's only a matter of time before Linux takes
its ...
techxav.com/.../my-review-of-10-reasons-why-linux-will-triumph-over-
windows/ -Cached - Similar
10. **** Jack Wallen - news ****
Jack Wallen asks Why aren't schools adopting open source? and the answer
is simply exposure to open source. Getting thousands of students to call
and write ...
www.wikio.co.uk/news/Jack+Wallen -Cached - Similar
Because these results will fly by you might want to do one more trick with this command by sending it to a file instead. To do this we will use the ">" character and a file name like so:
curl -A Mozilla http://www.google.com/search?q='Jack Wallen' |html2text -width 80 > search
Once the search is done, issue the command less search and you will see the results of your search.
Update Twitter status
The same kind of trick is done with your twitter status. To do this you would issue a command like:
curl -u user:password -d status='Your tweet' http://twitter.com/statuses/update.xml
Where 'Your tweet' is the status update you want to enter.
Right after you issue that command a bunch of information will be returned. This information is your profile rendered in XML. I have yet to find a way to turn that off.
Final thoughts
The command line is an incredibly powerful tool. Imagine what you can do with these two examples. You could set up a cron job to update your status automatically. Or you could create a script to automatically gather data from Google searches. The possibilites are limitless.
Advertisement
you could always do -o /dev/null on the update command to stop the output from showing
@k: Oh yeah….’doh on my part. ;-) thanks for that.
curl -A firefox http://www.google.com/search?q='Windows 7′ | html2text -width 80
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1350 100 1350 0 0 53143 0 –:–:– –:–:– –:–:– 1318k
Google Error
****** Bad Request ******
Your client has issued a malformed or illegal request.
whoops, forgot to add my question to that above
..
if i do it w/o the html2text cmd, it spits the html out of course, html2text isn’t converting something properly? (yes it’s installed).
Any idears?
If you paste the output minus any user/password information something like pastebin I can have a look at it
The reason is because your query string “?q=Windows 7”. For some reason it screws up google search. It must be some special keyword to do with numbers if you do “?q=Windows 1” the same thing occurs. You can get around it by doing “?q=Windows%207”, %20 is the hex value for space. You can use sed to convert lots of spaces something like:
echo “http://www.google.com/search?q=Windows 7 is what I want” | sed ‘s/ /%20/g’
then you could pipe that to curl using xargs by adding: | xargs -n1 curl -A mozilla
Hope this helps
word. thanks k. makes plenty of sense now (duh!), didn’t even think of that
It’s not obvious why “space[0-9]” causes a bug but I guess it has something to do with the commands google search runs eg. if you search for 1+1 it runs the calculator function. Must be something to do with that.
I recently added surfraw to cygwin and in my linux installations. Works great with links terminal browser for a number of searches. Plus less typing.
http://surfraw.alioth.debian.org/
For twitter, I was able to get termtter to work in cygwin. http://termtter.org. I use it nearly everyday now. Didn’t get it to work in ubuntu for some reason; might be how I installed ruby. A ghacks tutorial would be great. :)
Quitter a command line twitter client, can also be used for updating twitter from command line.
Goosh is another command line interface for Google search. So , ig you bassically love command line and cant leave windows, goosh should be the ideal choice for you.
You should do a blog about how one can capture a. Fine as hell band dudes heart.