The minds at IBM have decided the next challenge for one of their supercomputers will be to take on human competitors in the TV game show Jeopardy. It's been twelve long years since Deep Blue beat Garry Kasparov and the IBMers have decided to re-enter the world of high profile human-computer competition with a new machine, code-named "Watson".

Other than giving IBM bragging rights and plenty of free publicity, the technology developed for the Jeopardy challenge will have applications outside of the TV game show arena. The natural language ability of Watson will be useful in everything from Internet search engines to futuristic expert systems.

An article from the New York Times describes the purpose, challenges and controversies of the Jeopardy effort:

...but chess is a game of limits, with pieces that have clearly defined powers. “Jeopardy!” requires a program with the suppleness to weigh an almost infinite range of relationships and to make subtle comparisons and interpretations. The software must interact with humans on their own terms, and fast.
...
The team is aiming not at a true thinking machine but at a new class of software that can “understand” human questions and respond to them correctly. Such a program would have enormous economic implications.

Despite more than four decades of experimentation in artificial intelligence, scientists have made only modest progress until now toward building machines that can understand language and interact with humans.

The proposed contest is an effort by I.B.M. to prove that its researchers can make significant technical progress by picking “grand challenges” like its early chess foray. The new bid is based on three years of work by a team that has grown to 20 experts in fields like natural language processing, machine learning and information retrieval.
...
Under the rules of the match that the company has negotiated with the “Jeopardy!” producers, the computer will not have to emulate all human qualities. It will receive questions as electronic text. The human contestants will both see the text of each question and hear it spoken by the show’s host, Alex Trebek.

The computer will respond with a synthesized voice to answer questions and to choose follow-up categories. I.B.M. researchers said they planned to move a Blue Gene supercomputer to Los Angeles for the contest. To approximate the dimensions of the challenge faced by the human contestants, the computer will not be connected to the Internet, but will make its answers based on text that it has “read,” or processed and indexed, before the show.


There is some skepticism among researchers in the field about the effort. “To me it seems more like a demonstration than a grand challenge,” said Peter Norvig, a computer scientist who is director of research at Google. “This will explore lots of different capabilities, but it won’t change the way the field works.”
...
I.B.M. will not reveal precisely how large the system’s internal database would be. The actual amount of information could be a significant fraction of the Web now indexed by Google, but artificial intelligence researchers said that having access to more information would not be the most significant key to improving the system’s performance.

Eric Nyberg, a computer scientist at Carnegie Mellon University, is collaborating with I.B.M. on research to devise computing systems capable of answering questions that are not limited to specific topics. The real difficulty, Dr. Nyberg said, is not searching a database but getting the computer to understand what it should be searching for.

The system must be able to deal with analogies, puns, double entendres and relationships like size and location, all at lightning speed.

In a demonstration match here at the I.B.M. laboratory against two researchers recently, Watson appeared to be both aggressive and competent, but also made the occasional puzzling blunder.

For example, given the statement, “Bordered by Syria and Israel, this small country is only 135 miles long and 35 miles wide,” Watson beat its human competitors by quickly answering, “What is Lebanon?”

Moments later, however, the program stumbled when it decided it had high confidence that a “sheet” was a fruit.

The way to deal with such problems, Dr. Ferrucci said, is to improve the program’s ability to understand the way “Jeopardy!” clues are offered. The complexity of the challenge is underscored by the subtlety involved in capturing the exact meaning of a spoken sentence. For example, the sentence “I never said she stole my money” can have seven different meanings depending on which word is stressed.

The project-behind-the-project is DeepQA, a broader effort underway at IBM Research focused on the accessibility of natural language content and the advancement of natural language processing. From IBM's DeepQA site:

The DeepQA project at IBM shapes a grand challenge in Computer Science that aims to illustrate how the wide and growing accessibility of natural language content and the integration and advancement of Natural Language Processing, Information Retrieval, Machine Learning, Knowledge Representation and Reasoning — along with massively parallel computation — can drive open-domain automatic Question Answering technology to a point where it clearly and consistently rivals the best human performance.

Moving beyond the Jeopardy! competition, automatic Question Answering will help drive the future of business intelligence, analytics and information management, so that business and government decision makers will have the most cutting-edge capabilities for finding the precise information they need from the mountains of data they produce.

The site goes on to explain why the Jeopardy challenge is a perfect way to test DeepQA technology:

• Answering a Jeopardy! clue involves quickly focusing in on the most relevant parts of a clue, instantly identifying and dismissing non-essential or irrelevant parts, and then finding and weighing hundreds or thousands of bits and pieces of evidence to deliver a precise and correct answer.
• The best Jeopardy! players, according to our analysis, are able to do this over vast domains of knowledge and deliver the correct answer better than 85% of the time.
• If a player buzzes in and gets the question wrong, he or she is penalized. Winning, therefore, demands the ability to "know what you know" — to, in effect, determine an accurate confidence in whether or not you know the correct answer.
• While a natural and innate ability for humans, a challenge for a winning computer system is to consistently and very rapidly decide whether or not it has a correct understanding of the question and high enough confidence in its answer, based on the totality of what it has read, to take a chance on buzzing.