Where's Search Heading?

Contrarian investment strategy holds that when everybody is jumping on a particular bandwagon, it's time to sell. Shares of Google Inc. spent the last year trading in a range roughly between 450 to 550 with some modest but steady growth. Suddenly, about the time Merriam Webster's Dictionary recognized Google as a verb, shares shot up to over 700 ... briefly. It's not my intention to dispense investment advice, I just find that spurt of ill-founded exuberence interesting in light of the developments in the search-engine world.

I've been watching Google since their early days (at least since Nov 1999), and initially I was a big fan. The original PageRank algorithm was groundbreaking when it was introduced, and the name Google -- derived from Googol, the largest named number -- was an acknowledgment of the enormity of the task of indexing the World Wide Web (considerably less enormous then). They even had a cool motto: "Don't be Evil."

That Google is ancient history. Since their IPO in 2004, the professional management team at Google, Inc. has focussed far less on resisting evil in favor of short-term corporate profits. They are basically corporate raiders, buying up promising technology companies as opposed to fostering innovation. They bought Blogspot (which hosts these pages) and FeedBurner (which I use a lot) without much noticeable effect either way. Some of their other acquisitions and antics have recieved less favorable reviews.

Google is Microsoft's chief rival in developing (or acquiring) web-based applications, which both companies hope will be the "next big thing" in computing. In their efforts to broaden and deepen their reach, it seems that Google has let their flagship search engine go by the wayside. This blog was originally established primarily to counter Google's "thin affiliate" offensive (ca. June, 2005), wherein the search giant decided that merely organizing links in a logically significant fashion was unworthy of listing in their search engine. (Wait a minute! Isn't that what Google professes to do?)

Over the past three years, Google has repeatedly made their search narrower and shallower -- ostensibly improving relevance, but arguably just making their revenue-producing Ad Words program more competitive with their free search results. After about a year of this, according to data from Alexa, Google's reach peaked, and began to go downhill. Their response was to continue crippling (or "improving") their search function in new and more aggressive ways. The end result is that the once-powerful Google search engine now produces pretty lackluster results.

A few years ago, search engines engaged in size wars -- each claiming to index more pages than the other -- until critics began to make the extraordinary claim that size didn't matter. Of course size matters: you can't retreive pages that aren't indexed. What the critics should have said was that size alone was a poor indicator of search engine quality. Let's look at the approximate number of search results the three main search engines produce for some relevant websites as of today (2 APR 08):

Site:	Google	Live Search	Yahoo
Amazon.com	(43.9 M)	(75.1 M)	(448.5 M)
Amazon aStores	(424 K)	(1.96 M)	(19.6 M)
Wikipedia (English)	(3.8 M)	(31.2 M)	(191.4 M)

Search relevance is harder to evaluate. To begin with, the very definition of relevance is elusive. But assuming that all the major search engines make a reasonable good faith effort to organize their output based on commonly accepted industry practices, which search engine would you choose? According to a frequently-cited "recent" survey 58% choose Google. I guess it just goes to show, "You can fool some of the people all of the time."

An even greater irony is the fact that the current buzz in the search engine community is all about Yahoo! (See Where's Search Heading? Ask Yahoo's Chief Scientist) Is it any wonder Microsoft is trying to buy Yahoo outright?