« Demos from the Epic PAWS Meetup | Main | Wolfram Alpha: A Marketer's 2 Cents »

April 24, 2009


TrackBack URL for this entry:

Listed below are links to weblogs that reference How *not* to rate a search engine:



Great post Mark. I agree the simple search box completely belies what goes on underneath. It’s a disservice to the users at large when reviewers generalize five queries experience to billions of queries. Keep up the good work.

Daniel Tunkelang

Mark, nice post. But I do have to pick on Powerset here. I attended a Powerset presentation before the acquisition and asked about evaluation metrics--especially since my experience as a beta tester had been underwhelming. I didn't get a straight answer--and I haven't to this day. Were there ever any published experiments using TREC, user studies, or some other evaluation methodology? Or is all of the evaluation work still guarded as trade secret?


Spot on Mark. Many who review or rate a search engine's capabilities know little about how search works, they only know how results present. Even then, most compare it to what they already know (Google/Yahoo/Live/etc.) instead of removing their rose-colored glasses to look clearly at the advantages and disadvantages of something new. Obviously, the engine needs to perform for the user. But reviewers (often, media reviewers - not technical reviewers) judging engines based on just a couple searches shows the short-sightedness of the reviewer, and not necessarily that of the engine. Thanks for the post.

Avi Rappoport. SearchTools.com

I do agree with you. When I want an example for a talk or an article, I end up doing a bunch of searches to find just the right one to illustrate the point I'm trying to make. And if I can't find a good one, I re-examine my ideas.

But how many queries and what kind are enough to rate a search engine properly? Enterprise search isn't web search, and rarely has much control over the algorithm.

I have my own ideas, based on search log analysis, but most sites don't have the kind of traffic you are talking about. It turns out that a week's worth of search logs, if there are only 20,000 searches, have a very small head and a very long tail. And a whole lot of search spam and URL queries.

How I wish I had your mountains of queries and click data and money for human judges!

anniversary messages

I grace with your presence a Power set arrangement earlier than the achievement and asked on the subject of evaluation metrics.

michael kors outlet

Ha, ha, ha, good web site, the site of the construction of the really very good, let the first thing I love this website, and still he so brilliant, so rich connotation website, I will focus on such a web site, but also the good mood.

The comments to this entry are closed.