Deliberate Ambiguity

Death of the Search Startup

Death of the Search Startup

View more presentations from philosophygeek.

(note: this is a post based on a talk of the same name at Ignite SMX West)

Search is hard.  Really, hard.  As a veteran of three search engines, some might call me jaded.  However, this post isn’t just an attempt at apologetics.  I’ve learned something from those three experiences that I’d like to share with you.

If you’re looking to start a profitable business, it’s reasonable to wonder: can a startup ever pop up and beat Google?   As entrepreneurs, we always think that there’s a way to innovate our way out of a problem.  I’m going to argue that no amount of innovation can feasibly compete with Google or Bing.  Though marketing concerns like switching costs are certainly a problem for a search startup, I’m going to focus on an even deeper problem: the mighty greenback.  When it comes down to it, building a search engine is an incredibly expensive proposal.

In order to make some calculations, I created an equation that has two major components: hardware and people. I’ll give some explanations for each of the components and, in the process, show you how complicated and expensive a search engine is to build.

 

• Storage/Crawling (SC) – Two years ago, Google estimated the Web to be over 1 trillion documents and it’s growing every day.  A search engine needs to crawl and store the entire Web and keep frequently changing pages up-to-date.  Plus, your index is bigger than just Web pages.  You’ll be storing all sorts of metadata for the page: anchor text, outbound links, and any kind of interesting data you’ve extracted or created.
• Relevance (R) – At bare minimum, your search engine will have to have results as good as Bing or Google’s.  To achieve this, you’ll need servers to run relevance experiments, servers to store vast amounts of click-data, independent judges, and a big team to deal with all of the irrelevant/spammy sites out there.  And don’t think you’re done if you’ve just created 10 blue links.  Users have come to expect lots of other services: image search, stock quotes, weather answers, and news search, just to name a few.  You’re either going to have to license content or build verticals of your own. Both are expensive propositions.
• Runtime (RT) – When you do a search on Bing, you get a list of just 10 relevant results in fractions of a second from a set of possibly billions of Web pages.  How is that possible?  The short answer is: lots, and lots, and lots of computers to calculate your results.  Search engines typically use some kind of divide-and-conquer methodology, which means that whenever you issue a search, hundreds, or possibly thousands, of computers are involved in bringing you back results.
• People (P) – It’s unlikely, if not impossible, that you’re going to build a search engine with your buddy from college in a garage.  Worse, you need really expensive employees: PhDs in computer science, machine learning experts, statisticians, search engine veterans, infrastructure geniuses, etc. I estimate that it takes a bare minimum of 250 people to build a search engine. Yikes!
• Time (t) – Even with the smartest people, it’s going to take time to get everything running properly.  I estimate at least two years to get all of the components working together.
• Johnson Coefficient (Ĵ) – No equation is complete without an eponymous component.  However, I didn’t just stick in Ĵ because I love my last name.  The Johnson coefficient is a real concern for innovative search engines.  It’s all well-and-good if you’ve created a search engine that is equivalent to Bing or Google, but usually you want to make something better.  Unfortunately, better usually translates into more data and/or more processing . . . which translates into more machines.  Thus, the Johnson coefficient is the “tax” that innovative companies will pay for their innovation.

When you add it all up, my guesstimate for the bare minimum cost of a search engine is $100M dollars and could be as much as (or even more than?) $300M dollars.  Numbers like that, for a business that still hasn’t figured out how to switch people away from Google and Bing will terrify investors.

I have two case studies: Powerset and SearchMe.

Powerset can’t be seen as a complete failure, since Powerset was acquired by Microsoft.  At Powerset, all of our linguistic processing caused our index to be at least 10-20x bigger than a standard search engine.  Plus, the linguistic processing and matching we did at the runtime was much more expensive than typical search engine retrieval.  With about $25M injected into the company and 60 people, we were only able to index 2.5M documents of Wikipedia.  To have indexed the whole Web would have been much, much more expensive.

SearchMe was a search engine whose gimmick was to show thumbnails of Web pages in a fancy cover flow interface.  Imagine the cost of processing, storing, and serving all of those images.  It was no surprise to me when SearchMe shut its doors, since it claimed it needed another $50M in capital (!!!) to survive.

However, I encourage you to Keep Hope Alive!  Though I don’t think that another general purpose search engine could compete with Google, there’s a lot of opportunity on the periphery for special purpose search engines and vertical search engines.  And heck, if you’re lucky like Powerset, you might get bought by Bing or Google!

The moral of the story is that, next time you hear someone proclaiming how they are going to be the next Bing or Google, smile and nod, and keep your hard earned money far away from an investment in that ill-fated venutre.

How *not* to rate a search engine

While demoing Live Search at the Web 2.0 Expo, people continually asked the same questions: “What makes Live different?” or “Show me some features that will make me want to switch from my search engine” or the extremely confrontational “Why do you think you’re better than Google?”

My first instinct was to dive in and show people the coolest features in Live Search (e.g., demoing  Virtual Earth with an Xbox controller) or to let them play around with their own queries.

However, given my experience working for several startup search engines, I’ve come to realize that it's extremely difficult to convince someone that you’re better than another engine with words, features, or few carefully chosen queries.

So, after awhile, I started my demos with a caveat about the nature of a search engine: I implored my audience to try out Live Search for a week so that, in the words of the immortal Lavar Burton of Reading Rainbow, “But, you don’t have to take my word for it.”

Is this a cop-out?  Why is demoing search so hard?

Search “Features”

When showing off a new version of Microsoft Word or Typepad or Yahoo Messenger, a good product marketing person will not just demonstrate features, but analyze their audience and demonstrate benefits that help users accomplish specific tasks. (This is just product marketing basics.)

A search engine, by contrast, has an extremely simple interface: you type in some words and hope that the engine will cough up pointers to helpful Web sites or give you a direct answer.  The inner workings of a search engine, i.e. how those results were produced, are completely opaque to the user.  Hundreds of features are used to rank results so that the right Web sites and answers show up on a page when you type in some string of words.  Those features don't surfaces as demonstrable chunks that can be easily summarized or understood.

Common mistakes when evaluating a search solution

 Which brings me to the biggest mistake people make: judging a search engine by typing in a few queries and analyzing the results. There are many interrelated reasons that this methodology fails:

• A few good/bad results don’t mean that all results will be good/bad – even if you try out five searches and all are good, how do you know if your sixth is also going to be good?  That is, since you don’t know what is going on under the hood, you can’t make any predictions about the quality of future results.
• It’s hard to select a representative cross section of queries – people usually try out a few navigational queries, a vanity query, and a few queries that are either damn-near impossible or extremely obscure.  None of these sets represents an accurate cross-section of your monthly query log.
• What you think is “good” may not be good for the majority of users – for navigational queries (e.g. “CNN.com”) the top result is clear.  For more complicated queries, the top results are rarely obvious.
• Queries are out of context – we had this problem at SideStep all of the time.  During usability studies, users who were simply evaluating the look-and-feel of the product and scanning for cheap flights without any end-goal were never as good as users who were actually trying to buy a flight for a real trip.  A search engine should help you complete tasks, not just give you a pretty page or have links that look useful.
• People tend to focus on the first result – some queries just require one result.  But many queries should be judged by the diversity of interesting results.

There are probably countless other mistakes that are made during solo evaluations of search.   Therefore, search engines big and small realize that problems of ranking and relevance – the core of any search project – are solved only by lots and lots and lots of data from lots and lots and lots of people.  To solve this data problem, we need to collect data from real users.  For example, we run many thousands of queries past human judges and look at mountains of click data from the production site.  After applying apply advanced statistical techniques to this data, we get the information we need to create algorithms that turn your few (mispelled) words and turn them into a useful page of results.

As one of my colleagues at Powerset always likes to remind me: this is rocket science.