Deliberate Ambiguity

Recap of "Don't Call it a Comeback: Semantic Technology and Search" at SES SJ 2009

(c) Ken Yeung

Today I was at SES San Jose and represented Bing on a panel called "Don't Call it a Comeback: Semantic Technology and Search” hosted by Dana Todd.  The format was interesting: the four major search engines acted as the panel and three startups – Twine, WolframAlpha, and HubPages – gave presentations.

SES is very pragmatic conference focused on publisher issues, whereas semantic search discussions typically are forward-looking or, perhaps less generously, speculative.  Luckily for our audience, our conversation was peppered with practical examples.

Nova Spivack of Twine started off by arguing that the Web is now too big for keyword search. He reasoned both that users need more tools to organize the millions of results they get back for a query and that they want to ask more complex questions of that data. He suggested that the next steps in this third decade of the Web (since Tim Berners-Lee wrote his seminal paper in 1989) will be natural language search, semantic search (defined by Nova as “parameterized search”), which will eventually give rise (in Web 4.0) to applications that are able to make inferences and reason.

The most exciting portion of his presentation was his announcement of Twine 2.0: semantic search at Web scale.  He humbly admitted that Twine is a small company and that Twine 2.0 is an incredibly ambitious project (indeed!).  In a thinly veiled plea to the search engines on the panel, he suggested that he'd like to partner with one of us.  I hope to get a demo soon!

Next in line was Lane Soelberg from CoopVentures, who is helping to bring WolframAlpha to market (I wrote a post about WolframAlpha marketing prior to their launch).  Instead of just rehashing what WA does, he gave some compelling examples of how publishers could create mashups using the forthcoming WolframAlpha API and their own data.  One example was a hypothetical insurance company that combined their own internal accident database with WA weather, traffic, and holiday data to create an Accident Predictor.

Finally, Jason Menayan from HubPages gave a very practical example of how semantic technology can be used in SEO.  HubPages is a UGC site that generates over 60MM page views a month.  HubPages tried two experiments.  First, they semantically generated content suggestions for a few pages using Digger.  Though only done with 3 pages that hadn’t shown up in Google results yet, the results were positive.  The content suggestions broadened the topics on the page and caused them to be ranked highly within a few weeks.  HubPages is in the process of experimenting with more pages to get a more statistically significant set.  The second test was to present both semantic tags from Relevad and regularly generated tags as suggestions for the author of the page.  The idea of the semantic keywords is sometimes they surface relationships that are less likely to be found in a typical keyword style. Both sets of keywords seemed to help traffic equally as well up to a point -- the curve started to fall around 35 tags / per page. Not a clear win for semantics, but keep in mind that "folks" were still choosing the tags for the folksonomy.  Maybe automatic (semantic?) tag generation would be more efficient? .

Given the fascinating presentations, there wasn’t a lot of time for the folks from the search engines to comment.  We did get a few chances to respond to questions from Dana and the audience.

Oddly enough, the representatives from Google, Yahoo, and Bing (me!) were all people focused on creating the next generation of rich captions and had great statistics to share.  According to Kevin Haas from Yahoo, “lots and lots” of searches contain SearchMonkey results (try this).  After a bit of prying, he conceded that the number was over 50%, but his demeanor suggested that this number was much higher.  Personally, I’d be more interested in the number of results that have a SearchMonkey result in the top 10 or top 3 results, as a better measure. 

The first question from the audience came from a concerned audience member who worried that search engines could too much control of  content once the content was marked up.  Othar Hansson from Google assured the publisher that, after testing Rich Snippets for months, enhanced captions actually promoted click to the vendor sites.  He also stated that Google focuses on informing the user click and that they don’t believe many user queries can be answered on the SERP itself.  I humbly disagree, but will leave the judgment ultimately up to the users.

Another question to the search engines was: what are you doing in semantic search today?  In typical search engine style, we were all a bit cagey with our responses, partially because it’s hard to succinctly describe the technology that goes into any given search.  My response went something like this: given that Bing is a decision engine, it’s critical that we approach semantics from both the query side and the document side.  On the query side, a user should be able to type in a query in a way that’s most natural to her vocabulary, not the specialized vocabulary of a search engine.  Therefore, it’s incumbent upon us to translate the normal language of the user into the language of a search engine (if necessary).  However, the document side is just as important. The more a search engine knows about a document, the better we can rank the document in search results, the better snippet we can generate, and the more information we can present on the SERP.  Bing is certainly working to make strides on both fronts. Usually these advances are opaque to you, the end-user, and so much the better!  We’re all about solving user problems, not about touting our technical prowess.

I’d love to hear comments from readers, both on my impressions of the content of this panel and what you’d like to hear about on panels in the future.  No doubt there will be more semantic  search discussions at conferences in the future: what sorts of issues would you like addressed and what kind of burning questions do you have? Some questions that never got answered (courtesy of Dana Todd):

• Why has semantics struggled so hard for an identity, and what’s necessary to help it get market traction and mass appeal?
• What are the factors that limit/enable success for a semantic project or technology?
• We typically think of semantics in terms of words; WolframAlpha is now attempting to redefine to include numbers and pure data. How does that change things?

Happy Birthday Powerset.com!

It's hard for me to believe it, but a year ago today, we launched Powerset.com.  Of all the products I've launching in my career, I'm most proud to be a part of the Powerset team. Happy Birthday, Powerset.com!

So, how did it go? 

The launch itself was a resounding success.  Powerlabs and the amazing community (thanks, Powerlabbers!) helped to hone the final product into something highly usable, despite the underlying complex technology. 

Though we mistakenly flipped the switch and went live a few hours earlier than we had planned, the huge engineering effort to ensure launch capacity was worth it: Powerset went down only once for <30 minutes when we got slashdotted.

Press coverage was awesome.  The 30+ press briefings we did prior to the launch helped to calm some of the hype around Powerset ("No, we're not a Google-killer, yes, we have some cool technology, yes, this is a long journey," etc.).  Some of the hype is unavoidable, but I think people realized that we had the seeds of something really cool.

When we started getting real-world queries, we found that users are ready, willing, and able to type in natural language queries.  However, we also discovered that it's really hard to market a Wikipedia search engine to people (lots of people wanted to search for Pizza near their house).  As we predicted, once people had a taste of natural language search, they wanted much, much more.

Then, on July 1, 2008, it was announced that Microsoft had decided to purchase Powerset.

This was an amazing start, but the journey has just begun.  I wrote a post a year ago musing about whether Powerset crossed The Uncanny Valley.  I think it's safe to say that neither Powerset nor anyone after us has been able to create a semantic search experience that meets users' expectations. However, now that we're part of Microsoft, I'm really excited that Powerset will help to transform the Live Search experience.  We've got a lot of things in the pipeline, so keep checking back for more announcements.

Wolfram Alpha: A Marketer's 2 Cents

Wolfram Alpha will be launching on May 18, 2009, almost exactly a year after Powerset launched.  Unlike other big flops launches, I am totally stoked about the Wolfram Alpha launch, especially after reading all of great posts written about pre-demos.  In the spirit of nostalgia for Powerset's launch, I thought I'd toss my hat in a ring (unfortunately never having had a live demo), focusing on Wolfram Alpha's pre-launch from a marketing perspective.

Branding. In launching a new search service, a comparison to Google is inevitable.  We explicitly told reporters that Kosmix and Powerset were not competing with Google, but they cannont resist the siren song of a controversial headline.  Wolfram Alpha is wisely branding themselves as a computation engine, suggesting questions that crunch numbers instead of better search.  Also, despite comparisons to Google, Wolfram Alpha is trying to be humble and remind people that the road to computation is a long journey, not a quick win.  Wolfram's post The Quest for Computable Knowledge: A Longer View sets the right tone in a historical context.

Audience. There have been a lot of complaints that the name "Wolfram Alpha" is too long and complicated.  In demos, they've been showing off chemical formulas, population graphs, and calculus. This will not appeal to my mom.  Instead of going after consumers, they are going after geeks who love data.  Most geeks associate Worlfram with Mathematica (and the ueber-geeks associate him with NKS).  I like the name, for now.

Competitors. Notwithstanding comparisons to Google, I'm surprised that few have compared Wolfram Alpha to True Knowledge, which is currently live and can answer inferential questions like who are queen elizabeth ii's grandchildren and who was the first woman to climb mount everest and what time is it in Boston.  Note that both of these questions require inference, but TK knows about QEII's kids and their kids (and that kid's kids are grandchildren).  Unlike Wolfram Alpha, TK has a system for users to enter and curate information in their database.  If TrueKnowledge can get enough editors, this is potentially a much more flexible system than manually curating government data.  Another obvious competitor is Metaweb's Freebase. It'll be interesting to see who else pops up in this space.

Benefits.  This is where Wolfram Alpha is going to have some problems.  It's just not very often that I have questions about chemical formulas and when I type in "Seattle, WA" the graph of the temperature might be interesting, but probably not what I'm looking for.  What is it that I should use a "computation engine" for?  At least in the short term, they're going to rely on geeks getting lost in all of the cool things that Wolfram Alpha makes possible. 

Scope.  As a corrolary to the above, Wolfram Alpha better do a phenomenal job on day one of explaining what data is availabe and what kind of computations can be done on that data.  If they're smart, they'll have tons of example queries that show off the power of the engine, lots of screencast and tutorials, and maybe even a contest to get users to generate cool queries.  Based on what I've seen, here are a few demos that I'm gonna try (of course I'm going to take my own advice and not rate Wolfram Alpha based on the results):

• Chemical formula of water
• Number of moles in a gallon of water
• What happens when you mix bleach and ammonia?
• What states are in the Pacific Daylight time zone?
• What's the time difference between Tokyo and New York City?
• What animals have a cloven hoof?
• Is it colder in San Francisco or Seattle in May?
• When did the wooly mammoth live?
• What were the top 10 largest cities in the US in 1990?

*waiting with bated breath*

Demos from the Epic PAWS Meetup

Tonight I’m at a huge PAWS meetup with well over a 100 people (standing room only) here at the Microsoft campus in Mountain View (known to insiders as "SVC").  These are my notes from the presentation, so apologies if they look like a Virginia Woolf novel.

Uptake (Gene McKenna) - Has a vocabulary of travel related terms, sentiment analysis based on types of classes (positive/negative terms for romantic), an ontology (e.g., what does it mean to be kid-friendly?).  Then run reviews through their engine and extract interesting facts.   Indexed over a million products, 20 million opinions, etc.  For hotels, they show snippets of reviews that will be relevant, rather than have the user go through hundreds or thousands of reviews.  For example, a search for family hotels in San Francisco gives you a set of family-friendly hotels with highlights from the reviews. 

Expert Systems (Walter Pezzini) - FY08 $13.5M, profitable and 100+ employees(!!) and based in Italy.  Demo of Cogito Answers, which is answers NL questions from unstructred data (Wikipedia), some structure (eBay), or structured data (American Football DB).  This was developed for a customer and is not live on the Web. First question "who was OJ Simpson's wife?" got an answer from both Wikipedia.  Second question "do you have a Jets keychain for under 105?" gets results from eBay.  Third question: "for which teams did Farve play" went to the American Football DB.  The demo seems to show that Expert Systems can figure out which source to get the data from, but I would have been even more impressed if he asked a question that could have been answered from multiple sources to see how it collated the data.  Someone from the crowd: "Do you need to know the schema of the database?"  Answer: some tools for customers to map tables to concepts.

Freebase (Jason Douglas) - an (awesome) open, editable, free database with about 5M topics in the graph with their relationships.  Using the Metaweb Query Language (MQL), did a mashup with last.fm data + twitter follower data, to determine the most popular band member's name.  He should totally build a Web 2.5 mashup site and start his own company.

zAgile (Andrew Lampitt) - Why semantic Web in the enterprise: heterogeneous, siloed apps + no shared vocabularies + no resuse across apps and processes.  Demonstrates how Wikidsmart (which is the best product name I've seen this millenium) can help to integrate across many different software development apps (version control, bug tracking, project managment, etc.) in a nice Wiki interface.  You can try out their sandbox once you log in.

Zemanta (Bostjan Spetic aka Bos aka Zemanta co-founder) - takes a few lines of text, recognizes what's important, pulls back pictures relevant to the text, suggests links from multiple sources, and other pieces of information. It's easy, just copy some text and try Zemanta here.  Think "research assistant" or "content enhancement."  Obviously sits on a lot of technology to do this, but tries to hide the complexity from the user: "so you and your mother can use it."  All of the processing is done on Zemanta's servers. You can use Zemanta by installing it as a plug-in and it will work in your Gmail, Yahoo Mail, Wordpress blog, Typepad blog, etc.  Also has an API that allows you to do Semantic Mashups.  

Noovo (Andrej Babergoj) - Noovo is a recommendation engine in Beta that suggests content and tell you why it recommended it and who recommended it.  You can give items a thumbs up or down. (This was a real demo: one of his friends shared an NC-17 picture with him.  Friggin' awesome.)  Allows you to join groups, connect with your social networks, etc. - things you'd expect in a Web 2.0 recommendation engine. Social graph (who are your friends/colleagues/family) + Activity graph (what you do) + Personal Interests (what you like).  Someone asked a question (which was also on my mind) about how they compare to Twine.  I didn't completely understand the answer =)

Microsoft Research (Bora Beran) - showing off some work done at MSR, specifically in the Technical Computing group.  Showed off a few plug-ins for Microsoft Word.  First, Chem4Word allows you to all sorts of features to add chemicals to Word Docs.  For example, he typed in "water" and got a list of synonyms and was able to insert an dihydrogen monoxide molecule into the document.  Zentity allows you to manage your scholarly work (not using OWL or RDF, but an identity data model) for things like citations and gives you an advanced way to search through the data.  SciScope is (one of) Bora's project, which is an interatcive map with tons of environmental data.  Takes data from multiple data sources, but totally transparent to the user.

One last special mention: I met Guha Jayachandran, who did Folding@Home as his PhD project at Stanford.  He's got a new search engine called Cruxlux which analyzes perspective.  Very excited to see that.

Let me know if there are any errors in my trascriptions.  And follow me on Twitter!

Overview of Semantic Search Engine Panel

As previously reported, I was lucky enough to host a panel at Alternative Search Engine Day last Monday.  I thought I’d type up some of my notes from the panel and talk about what I thought was most interesting.

For the demos, we got a sneak peek into TrueKnowledge, which is still behind a closed beta.  TK currently houses over 106M facts and is constantly growing.  TK does inference, which means that it can derive facts from knowledge it has cataloged and knowledge it has about the world.  For example, TK might not have an explicit fact for Queen Elizabeth II’s grand-daughters, but it knows her children and their children and can derive the fact.  Users can query the system in natural language and, when a fact isn’t returned, can add that knowledge to TK.

Truvert is a vertical search engine for the green market (a popular topic at Alt Search Engine Day) which his built on technology from OrcaTec.  When a user types in an ambiguous term like “palm” into Truevert, the results are centered around “palm oil,” not the PDA.  OrcaTec claims that it can build a vertical with a small training set of a few thousand documents in just a few hours on a single machine.  Their search is based on Yahoo BOSS.

I was especially excited to see TextDigger, which I’d never seen demoed live before.   TD is currently in private beta with about 1000 users and a long waiting list.  Launch is anticipated in Q3 of this year.  TextDigger is meant to be a high-end research tool where users have lot of power over the semantics of their search.  TextDigger shows all of the possible senses of words in a given query and the user can select which sense they really meant, which will filter the search results accordingly.  I found this especially interesting because it’s a feature we always considered at Powerset, but ended up leaving out of the final product because of concerns about confusion of users (and that we thought that the heavy lifting should be done by the engine, not the end user).  I’m really excited to get an invite to the beta and I’m sure I’ll have a more detailed analysis once I get in and play around.

During the Q&A session, a number of interesting topics came up.  Both TextDigger and Truevert both rely on results from a bigger search engine and rerank them based on semantic algorithms.  TrueKnowledge has built their own index of facts, which is focused on structured information as opposed to free text.

When talking about what “semantic search” is defined as, Dr. Musgrove of TextDigger delineated a difference between “soft” and “hard” semantic search (reminiscent of strong vs. weak AI and the hard vs. easy problem of consciousness).  Better query suggestions, similar to what Google recently announced as soft, whereas actually trying to understand queries and read text is hard.  The distinction still isn’t completely clear in my mind, though my instinct suggests that it would be useful to discover.  I’m going to noodle that and see if I can come up with something.

Overall, great panel and I look forward to the upcoming Semantic Demo Session on 4/14 at Microsoft in Mountain View, Web 3.0, and the Semantic Technology Conference.

The Rising Stars of the Semantic Web (SemTech2008 Keynote Panel)

Here's my verbatim-ish transcription of the keynote at the Semantic Technology Conference in San Jose: The Rising Stars of the Semantic Web.

Powerset (Dr. Barney Pell, founder & CTO)

Powerset is organizing and indexing content from Wikipedia and Freebase.  First query is Henry VIII.  For topical queries, Powerset provides a dossier of.  Factz: automatically extracted triples from across Wikipedia articles.  For a famous person like Henry VIII, there are assertions all over Wikipedia about Henry VIII.  Second query: what did the fda approve?  Powerset doesn’t have any special domain knowledge (English history or medicine) that allows it to pull out information from text.  Search results are taking advantage of both semantic features and traditional search features. Now off to articles: Tim Berners-Lee.  Powerset republishes Wikipedia articles with some additional features, like the Article Ouline that follows you as you browse.  By looking at the Explore Factz, you can see all of the concepts in the page and the relations.  Clicking on a concept shows all of the Factz in the Article Outline.  Final query: what did tom cruise star in?  An NL query translated into a Freebase query, which shows all of the movies that Tom Cruise starred in. 

AdaptiveBlue (Alex Iskold, founder & CEO)

A network on top of the existing Web.   Basic bits of the web are understood by AB’s technology, e.g. books, stocks, addresses, etc.  Stop visualizing the Web as a collection of pages but as a collection of things.  For example, "The Kite Runner" may show up on both Amazon and Barnes&Noble.com, but they both refer to the same book.  Why is this different?  Bringing semantics to the Web as it exists now, not creating a new semantic Web.  This allows the content to evolve with the Web itself.  Two primary products are BlueOrganizer (a browser plug-in) and SmartLinks (semantic improvements to links on your blog).

Twine by Radar Networks (Nova Spivack, founder & CEO)

Actually this was a video made by a high school student who is now an intern at Radar Networks.  The video is to the tune of Radar Love and a fast-paced tour through Twine.  Unfortunately, there was so much going on that my limited human brain couldn't absorb it all: making new Twines, e-mailing things to a Twine, adding Google videos to Twine, history of video games, oh my!  Kids these days.  They just click them mice and keyboardz so fast.  Video ends with "Twine: How have we lived without it."

Talis (Ian Davis, CTO and creator of RSS 1.0)

Privately owned, well-established and innovative software company that's been around for 40 years,> all of the other companies on the panel combined.  Talis wants to be critical infrastructure to the semantic Web, such as applications for libraries, education, learning and community.  

Stealth-Company.com (Tom Gruber, CTO and founder)

No particular product to pitch ('cause they're still a stealth company).  The killer app isn't search, maps, e-mail, videos, etc.  It's intelligent design that brings together all of that cool stuff.  The iPhone is an example of this.  The killer app is your life (online): applying the best of the internet technologies (intelligently) to your life.  It's closer than we think (um gottes willen).  Paradigm shift.  Paradigm 1 is the hyperlink (user finds links, computer connects).  Paradigm 2 is the portal (user chooses your channels, computer delivers).  Paradigm 3 is the search engine (user states the query, computer finds relevant documents).  But why do we need to find the "magic" words to find what we want?  The next level will anticipate our needs.  I'm anticipating Stealth Company!

Q&A Session

• "What is the best opportunity that would get you a quote from Carla Thompson in the New York Times"
• • Nova - unsexy=a store of trillions of triples  sexy=x-rated "sementic" application
  • Alex - intersection of iPhone and the semantic Web, i.e. an application that understands your contexts as you move around
  • Tom - "the interface" companies need to address basic human needs.
  • Ian - similar answer to above.  Concrete example: something around travel.
  • Barney - search advertising and publishing.
• "Once you get Carla's attention, what do you do next.  How do you get to Gartner, Jupiter, and Forrester to notice?"
• • Alex - 2008 is the year of the semantic web, with or without Garner. 
  • Barney - "I think it's perfectly adequate that Guidewire is here: we don't need Gartner."  A big question is: what is the category that we're creating?  For example, what happens when we're able to join different pieces of content. . . new interface paradigms. . . new ways of consuming information. 2 or 3 categories will be created by the time of this conference next year.  "This conference next year will be twice the size and half of the companies will be consumer companies."
  • Nova - these firms don't think of "semantic web" as a space.  They just want to know how it's going to help the categories that they already track.  Maybe the name "semantic web" isn't really a good term anyway, since there are so many different companies, products, and underlying technologies.  Big analysts are playing catch-up.  The people in this room should define the category and educate the market.

Whew!  That was fun :)

Report from SemTech 2008 & the SD Forum Semantic Web SIG

Carla already did a great writeup of the panel I was on yesterday, so I won't bother taping out my disjointed thoughts.  However, I was at the SD Forum Semantic Web SIG panel last night (hosted at the Semantic Technology Conference) entitled, Will Semantics Give Web Search a Face Lift.

A common theme at this conference seems to be on the semantics of "semantic," and I found the problem to be especially apparent on this panel.  Oddly enough, there were no demos, but here's a summary of the panelists' opinions.  Consider the following:

• Healthline (Dr. AJ Chen) - since Dr. Chen was moderating, his brief slides focused on what a semantic search solution looks like under the hood.  Specifically, Healthline has health-specific ontologies that they're using for refinement, improved results, and improved ads.  Though I think Healthline's solution is very intersting, this is a pretty much the standard ontology implementation for vertical search companies. Semantic web = vertical ontologies used in a vertical domain.
• Google (Dr. Fernando Pereira) - Dr. Pereira seemed to have been sufficiently brainwashed by Google's Borg culture.  Though he admitted that Google is looking into next-generation technologies, his slides were a laundry list of why-not reasons: lexical information always lags behind real world usage, ontologies are messy, categorization schemes are difficult (impossible?) to create, annotations are noisy and potential spam channels. He cited utilitarian philosophy as Google's mantra: the greatest good for the greatest number of people.  Semantic web = interesting ideas, but I don't care unless it shows a marked improvement in horizontal results and can scale immediately to the entire Web.
• Yahoo (Dr. Peter Mika) - this talk was the closest to the standard W3C version of the Semantic Web.  Through Search Monkey, Yahoo is enabling content publishers to produce rich abstracts in their search results.  The idea is to provide a better user experience in search results, allowing users to get from "do to done faster."  Semantic Web = encouraging publishers to use standards and using them to improve search result listings.
• Hakia (Dr. Christian Hempelmann) - Dr. Hempelmann's slides began with a sigh about needing beer.  Fair enough.  His slides then talked about what he doesn't consider semantics: proximity, syntax, 80% solutions, small incremental improvements.  In contrast, he argued that Hakia's extensive ontology was the only way to get to true semantics. Semantic Web = our way or the highway.

Wow!  Four people on a panel, essentially talking past each other. On one end of the spectrum, Google seems to poo-poo semantics and NLP, generally. On the other side, Hakia seemed to be promoting a dogmatic view of what semantics is supposed to be.  Yahoo sits somewhere in the middle, trying to use approved technologies to improve results incrementally.  Healthline uses targeted ontologies to improve their vertical, but with little chance to improve Web search generally.

Luckily, Dr. Ron Kaplan, our beloved Chief Science Officer from Powerset, made some important comments.  First, he noted that his (admittedly biased) version of the Semantic Web could be called the "Syntactic Web," i.e., that structural relationships are a necessary condition for finding meaning.  He worried that some of the panelists were saying "Search is good enough. Semantics can't get us to God's Own search engine.  Therefore, Semantics isn't worthwhile."  Ron pointed out that if semantics can get us better search results, the real question is whether it's good enough for users to appreciate the difference without them rebelling because it isn't perfect.  My favorite quote related to this point was: "We know that stupidity scales.  That's not the problem.  The question is: can we do better than that?" (thanks to Uldis for capturing this on Twitter)

My Powerset-influenced opinion is that either extreme is wrong; that a general solution over incremental vertical improvements is key to overhauling Web search; that being open is better than being dogmatic; that a "semantic" web search will include semantics, syntax, statistics, and god knows what else; and that none of the major players seems are outwardly focused on this direction.

*Update* - Reposted on AltSearchEngines.

Powerset crosses the Uncanny Valley

In case you haven't heard the news, Powerset launched today.  Since I can't sleep due to launch adrenaline, I suppose it's time to weigh in on Powerset unofficially.  It's fun to write a press release, but it's more fun to philosophize on my own time!

Last year, I came across the concept of the Uncanny Valley though an employee of PARC.  The basic concept runs something like this: as you make a robot more like a human being, people tend to respond with more empathy.  However, there's a dip in the empathy curve when the robot looks enough like a human, but has behavior incongruent with what we'd expect from a real human being.  We tend to forgive weird mannerisms (jerkiness, oddities in speech, etc.) when a robot looks like a robot.  When a robot looks like a human, we expect it to act like a human.

As an example, cartoon characters are usually more believable than computer generated characters, unless the CG is so perfect as to be indistinguishable from reality.

Keyword search engines sit somewhere on the left of the curve.  We're willing to accept the choppy "keywordese" language of a search engine because we appreciate the results that come back and we can make excuses when it goes awry: "It's just a computer.  It doesn't really understand me."  As with most technology products, the utility curve and the expectation curve are aligned exactly because we treat keyword search engines as an unfeeling technology.

Oddly enough, when a system is endowed with an understanding of meaning (call it "semantics"), we suddenly demand that the system act, think, infer, and comprehend just like another human.  Instead of just being a cold of technology, we begin to treat the system with agency and want it to act like other agents (i.e., humans). 

The question shouldn't (and can't) be about releasing a system that has a perfect understanding of language, because that perfect understanding is many years down the road.  How is it that companies like Radar, TrueKnowledge, and Powerset believe that we can create products based on imperfect technology?

There are two ways to stay out of the Uncanny Valley.  First, create unique features that are so compelling that users are willing to deal with imperfection.  Keyword search hasn't survived because we always get the right answer - plenty of searches yield nothing.  But, though we may get wrong answers, we often enough get answers that would never have been possible before keyword search.  Second, access to semantic features shouldn't require a significant change in user behavior.  No matter how good (or "natural") a feature is, if a user has trouble finding it, it won't get used.

People often think that NL search is doomed to failure both because computers don't understand enough and, even if they did, users won't start entering natural language queries.

I think Powerset has scored on both counts.  As an example, for topical queries like moses or machiavelli, Powerset assembles a summary of Factz based on our deep linguistic analysis of each sentence.  This kind of automatic aggregation of content just wasn't possible before Powerset.  But, users don't have to do anything special.  They just get Factz "for free" by using their keyword usage paradigm.

I'm really proud of Powerset's first product.  The debate on whether or not we've crossed the valley will continue, but one thing is certain: a better, more natural way of interacting with technology is coming. (Of course, whether we sit to the left or the right of the Uncanny Valley is the subject for another post.)

Bed time, finally!

Biding my Twine

Damn it, I want in!  I haven't found a good program to manage all my life's text 'cause I'm too lazy, e.g. my anemic del.icio.us account.  My 'puter should just know why I tagged a page.  I sure as hell shouldn't be the one picking a word to place the article on the correct branch of my "folksonomy," because that graph is going to be an unholy orgy of Familienähnlichkeit.

Chatter about the Semantic Web will hopefully increase (Google trends indicates a decrease) as more companies, like Powerset and Radar, launch cool, useful applications.  Once users realize what's possible, they'll demand more from those indolent hunks of silicon sitting on their laps.