Can a Computer Win on 'Jeopardy'?

By STEPHEN BAKER

Developed over four years at an estimated cost of more than $30 million, IBM's "Jeopardy"-playing computer, Watson, will face the quiz show's grand masters, Ken Jennings and Brad Rutter, in two games to be aired Feb. 14, 15 and 16. As Stephen Baker relates in the following excerpt from his new book, "Final Jeopardy: Man vs. Machine and the Quest to Know Everything," doubts remain about how well Watson can process the endless subtleties of human language.

Watson paused. The closest thing it had to a face, a glowing orb on a flat-panel screen, turned from forest green to a dark shade of blue. Filaments of yellow and red streamed steadily across it, like the paths of jets circumnavigating the globe. This pattern represented a state of quiet anticipation as the supercomputer awaited the next clue.

Interactive: Match Wits With Watson

It was a September morning in 2010 at IBM Research, in the hills north of New York City, and the computer, known as Watson, was annihilating two humans, both champion-caliber players, in practice rounds of the knowledge game of "Jeopardy." Within months, it would be playing the game on national TV in a million-dollar man vs. machine match-up against two of the show's all-time greats.

As Todd Crain, an actor and the host of these test games, started to read the next clue, the filaments on Watson's display began to jag and tremble. Watson was thinking—or coming as close to it as a computer could. The $1,600 clue, in a category called "The eyes have it," read: "This facial wear made Israel's Moshe Dayan instantly recognizable world-wide."

The three players—two human and one electronic—could read the words as soon as they appeared on the big "Jeopardy" board. But they had to wait for Mr. Crain to read the entire clue before buzzing. That was the rule. At the moment the host pronounced the last word, a light would signal that contestants could buzz. The first to hit the button could win $1,600 with the right answer—or lose the same amount with a wrong one. (In these test matches, they were playing with funny money.)

This pause for reading gave Watson three or four seconds to hunt down the answer. The first step was to figure out what the clue meant. One of its programs promptly picked apart the grammar of the sentence, identifying the verbs, objects and key words. In another section of its cluster of computers, research focused on Moshe Dayan. Was this a person? A place in Israel? Perhaps a holy site?

In a room behind the podium, visible through a horizontal window, Watson's complex of computers churned, and the fans cooling them roared. This time, its three seconds of exertion paid off. Watson had come up with a response. The computer sent a signal to a mechanical device on the podium. It was the size of a large aspirin bottle with a clear plastic covering. Inside was a buzzer. About one one-hundredth of a second later, a metal finger inside this contraption shot downward, pressing the button.

Justin Bernbach, a 38-year-old from Brooklyn, stood to Watson's left. The airline lobbyist had pocketed $155,000 while winning seven straight "Jeopardy" matches in 2009. Unlike Watson, Mr. Bernbach understood the sentence. He knew precisely who Moshe Dayan was as soon as he saw the clue, and he carried an image of the Israeli leader in his mind. He had the answer. He gripped the buzzer in his fist and frantically pressed it four or five times as the light came on.

But Watson had arrived first.

"Watson?" said Mr. Crain.

The computer's amiable male voice arranged the answer, as "Jeopardy" demands, in the form of a question. "What is eye patch?" it said.

"Very good," Mr. Crain said. "An eye patch on his left eye. Choose again, Watson."

Mr. Bernbach slumped at his podium. This match with the machine wasn't going well.

***

It was going magnificently for David Ferrucci. As the chief scientist of the team developing the "Jeopardy"-playing computer, Mr. Ferrucci was feeling vindicated. Only three years earlier, the suggestion that a computer might match wits and word skills with human champions in "Jeopardy" sparked opposition bordering on ridicule in the halls of IBM Research. And the final goal of the venture, a nationally televised match against two "Jeopardy" legends, Ken Jennings and Brad Rutter, seemed risky to some, a bit déclassé to others. "Jeopardy," a TV quiz show, appeared to lack the timeless cachet of chess, which IBM computers had mastered a decade earlier.

Nonetheless, Mr. Ferrucci and his team went ahead and built their machine. Months earlier, It had fared well in a set of test matches. But the games revealed flaws in the machine's logic and game strategy. It was a good player, but to beat Messrs. Jennings and Rutter, who would be jousting for a million-dollar top prize, it would have to be great. So they had worked long hours over the summer to revamp Watson. This September event was the coming-out party for Watson 2.0. It was the first of 50 test matches against a higher level of competitor: humans, like Justin Bernbach, who had won enough matches to compete in the show's Tournament of Champions.

Watson, in these early matches, was having its way with them. Mr. Ferrucci, monitoring the matches from a crowded observation booth, was all smiles. Keen to promote its "Jeopardy"-playing phenom, IBM's advertising agency, Ogilvy & Mather, had hired a film crew to follow Mr. Ferrucci's team and capture the drama of this opening round of championship matches. The observation room was packed with cameras. Microphones on long booms recorded the back and forth of engineers as they discussed algorithms and Watson's response time, known as latency.

It was almost as if Watson, like a human giddy with hubris, was primed for a fall. The computer certainly had its weaknesses. Even when functioning smoothly, it would commit its share of wacky mistakes. Right before the lunch break, one clue read, "The inspiration for this title object in a novel and a 1957 movie actually spanned the Mae Khlung." Now, it would be reasonable for a computer to miss "The Bridge Over the River Kwai," especially since the actual river has a different name. Perhaps Watson had trouble understanding the sentence, which was convoluted, even for humans. But how did the computer land upon its outlandish response, "What is Kafka?" Mr. Ferrucci didn't know. Those things happened, but Watson still won the two morning matches.

It was after lunch that things deteriorated. Mr. Bernbach, so frustrated in the morning, started to beat Watson to the buzzer. Meanwhile, the computer was making risky bets and flubbing entire categories of clues. Defeat, which seemed so remote in the morning, was now just one lost bet away. This came in the fourth match. Watson was winning by $4,000 when it stumbled on this final clue: "On Feb. 8, 2010, the headline in a major newspaper in this city read: 'Amen! After 43 years, our prayers are answered.'" Watson missed the reference to the previous day's Super Bowl, won by the New Orleans Saints. It bet $23,000 on Chicago. Mr. Bernbach also botched the clue, guessing New York. But he bet less than Watson, which made him the first human to defeat the revamped machine. He pumped his fist.

In the sixth and last match of the day, Watson trailed Mr. Bernbach, $16,200 to $21,000. The computer landed on a Daily Double, which meant it could bet everything it had on nailing the clue. It was under the category "Colleges and Universities." A $5,000 bet would have brought Watson into a tie with Mr. Bernbach. A larger bet, while risky, could have catapulted the computer toward victory. "I'll take five," Watson said.

Five. Not $5,000, not $500. Five measly dollars of funny money. The engineers in the observation booth were stunned. But they kept quieter than usual, since cameras were rolling.

Then Watson crashed. It occurred at some point between placing that lowly bet and attempting to answer a clue about the first Catholic college in Washington. Watson's "front end," its voice and avatar, were waiting for its thousands of processors, or "back end," to deliver an answer. It received nothing. Anticipating these situations, the engineers had prepared Watson with set phrases. "Sorry," Watson said, reciting one of them, "I'm stumped." Its avatar displayed a dark blue circle with a single filament orbiting mournfully in the antarctic latitudes.

What to do? Everyone had ideas. Maybe they should finish the game with an older version of Watson. Or perhaps they could hook up Watson to another up-to-date version of the program at the company's Hawthorne labs, six miles down the road. But some worried that a remote connection would slow Watson's response time, causing it to lose more often on the buzz. In the end, as often happens with computers, a reboot brought the hulking "Jeopardy" machine back to life. But Mr. Ferrucci and his team got an all-too-vivid reminder that their "Jeopardy" player, even as it prepared for a national TV debut, could go haywire at any moment. When Watson was lifted to the podium, facing banks of TV lights, it was anybody's guess how the computer would perform.

***

Only four years earlier, in 2006, Watson was a prohibitive long shot, not just to win at "Jeopardy," but even to be built. For more than a year, the head of IBM Research, a physicist named Paul Horn, had been pressing different teams at the company to pursue a "Jeopardy"-playing machine. The way Mr. Horn saw it, IBM had triumphed in 1997 with its chess challenge. The company's machine, Deep Blue, had defeated the reigning world champion, Garry Kasparov. This burnished IBM's reputation among the global computing elite while demonstrating to the world that computers could rival humans in certain domains associated with intelligence.

That triumph had left IBM's top executives hungry for an encore. Mr. Horn felt the pressure. But what could the researchers get a computer to do? Deep Blue had rifled through millions of scenarios per second, calculated probabilities, and made winning moves. But it had skipped the far more complex domain of words. This, Mr. Horn thought, was where the next challenge would be. The next computer should charge into the vast expanse of human language and knowledge. For the test, Mr. Horn settled on "Jeopardy." The quiz show, which debuted in 1964, attracted some nine million viewers every weeknight. It was the closest thing in the United States to a knowledge franchise. "People associated it with intelligence," Mr. Horn later said.

There was one small problem. For months, he couldn't get any takers. "Jeopardy," with its puns and strangely phrased clues, seemed too hard for a computer. IBM already had teams building machines to answer questions, and their performance, in speed and precision, came nowhere close to even a moderately informed human. How could the next machine grow so much smarter?

Mr. Horn eventually enticed David Ferrucci and his team to pursue his vision. An expert in Artificial Intelligence, Mr. Ferrucci had a wide and ranging intellect. He was comfortable conversing about everything from the details of computational linguistics to the evolution of life on Earth and the nature of human thought. This made him an ideal ambassador for a "Jeopardy"-playing machine. After all, his project would raises all sorts of issues, and fears, about the role of brainy machines in society. Would they compete for jobs? Could they establish their own agendas, like the infamous computer, HAL, in "2001: A Space Odyssey," and take control? What was the future of knowledge and intelligence, and how would brains and machines divvy up the cognitive work?

For humans, knowledge is an entire universe, a welter of sensations and memories, desires, facts, skills, songs and images, words, hopes, fears and regrets, not to mention love. But for those hoping to build intelligent machines, it has to be simpler. Broadly speaking, it falls into three categories: sensory input, ideas and symbols.

Consider the color blue. It's something that computers and people alike can perceive, each in their own fashion. Sensory perception is the raw material of knowledge. Now think of the three-letter word "sky." Those letters are a symbol for the biggest piece of blue in our world. Computers can handle such symbols. But how about this snippet from Lord Byron? "Friendship is love without his wings." That sentence represents the third realm of knowledge: ideas. How can a machine make sense of these? In these early years of the 21st century, ideas remain the dominion of humans—and the frontier for thinking machines.

Over the next four years, Mr. Ferrucci set about creating a world in which people and their machines often appeared to switch roles. He didn't know, he later said, whether humans would ever be able to "create a sentient being." But when he looked at fellow humans through the eyes of a computer scientist, he saw patterns of behaviors that often appeared to be pre-programmed: the zombie-like commutes, the near-identical routines, from tooth-brushing to feeding the animals, the retreat to the same chair, the hand reaching for the TV remote. "It's more interesting," he said, "when humans delve inside themselves and say, 'Why am I doing this? And why is it relevant and important to be human?' "

His machine, if successful, would nudge people toward that line of inquiry. Even with an avatar for a face and a robotic voice, the "Jeopardy" machine would invite comparisons to the other two contestants on the stage. This was inevitable. And whether it won or lost on a winter evening in 2011, the computer might lead millions of spectators to rethink the nature, and probe the potential, of their own humanity.