Can a Computer Win on 'Jeopardy'?

Chapter Eleven: The Match



David Ferrucci had driven the same stretch hundreds of times. It was the route from his suburban home to IBM’s Yorktown labs, or a bit farther to Hawthorne. For fifteen or twenty minutes along the Taconic Parkway each morning and evening, he went over his seemingly endless to-do list. How could his team boost Watson’s fact-checking in Final Jeopardy? Could any fix ensure that the machine's bizarre speech defect would never return? Was the pun-detection algorithm performing up to par? There were always more details to focus on, plenty to fuel both perfectionism and paranoia--and Ferrucci had a healthy measure of both.

But this January morning was different. As he drove past frozen fields and forests, the pine trees heavy with fresh snow, all of the to-do lists were history. After four years, his team’s work was over. Within hours, Watson alone would be facing Ken Jennings and Brad Rutter, with Ferrucci and the machine’s other human trainers reduced to spectators. Ferrucci felt his eyes well up. “My whole team would be judged by this one game,” he said later. “That’s what killed me.”


The day before, at a jam-packed press conference, IBM had unveiled Watson to the world. The event took place on a glittering new Jeopardy set mounted over the previous two weeks by an army of nearly 100 workers. It resembled the set in Culver City: the same jumbo game board to the left, the contestants' lecterns to the right, with Alex Trebek's podium in the middle. In front was a long table for Jeopardy officials, where Harry Friedman would sit, Rocky Schmidt to his side, followed by a line of writers and judges, all of them equipped with monitors, phones, and a pile of old-fashioned reference books. All of the pieces were in place. But this east coast version was plastered with IBM branding. The shimmering blue wall bore the company’s historic slogan, Think, in a number of languages. Stretched across the shiny black floor was a logo that looked at first like Batman’s emblem. But closer study revealed the planet earth, with each of the continents bulging, as if painted by Fernando Botero. This was Chubby Planet, the symbol of IBM’s Smarter Planet campaign, and the modal for Watson’s avatar. In the negotiations with Jeopardy over the past two years, IBM had lost out time and again on promotional guarantees. It had seemed that Harry Friedman and his team held all the cards. But now that the match was assured, and on Big Blue's home turf, not a single branding opportunity would be squandered.

The highlight of the press event came when Jennings and Rutter strode across the stage for a five-minute, 15-clue demonstration. In this test run, Watson had held its own. In fact, it had ended the session ahead of Jennings, $4,400 to $3,400. Rutter trailed with $1,200. Within hours, online headlines proclaimed that Watson had vanquished the humans. It was as if the game had already been won.

If only this were true. The demo match featured just a handful of clues and included no Final Jeopardy--Watson’s Achilles heel. What’s more, after the press emptied the auditorium that afternoon, Watson and the human champs went on to finish that game and play another round--"loosening their thumbs," in the language of Jeopardy. In these games Ferrucci saw a potential problem: Ken Jennings. It was clear, he said, that Jennings had prepped heavily for the match. He had a sense of Watson's vulnerabilities and an aggressive betting strategy specially honed for the machine.  Brad Rutter was another matter altogether. Starting out, Ferrucci’s team had been more concerned about Rutter than Jennings. His speed on the buzzer was the stuff of legend. Yet he appeared relaxed, almost too relaxed, as if he could barely be bothered to buzz. Was he saving his best stuff for the match?


In the first of the two practice games, Jennings landed on all three daily doubles. Each time he bet nearly everything he had. This was the same strategy Greg Lindsay had followed to great effect in three sparring games 10 months earlier. The rationale was simple. Even with its mechanical finger slowing it down by a few milliseconds, Watson was lightening fast on the buzzer. The machine was likely to win more than its share of the regular Jeopardy clues. So the best chance for humans was to pump up their winnings on the four clues that hinged on betting, not buzzing. Those were the three Daily Doubles hiding behind certain clues, and the Final Jeopardy. Thanks to his aggressive betting, Jennings ended the first full practice game with some $50,000, a length ahead of Watson, which scored $39,000. Jennings was fired up. When he clinched the match, he pointed to the computer and exclaimed, “Game over!” Rutter finished a distant third, with about $10,000. In the second game, Jennings and Watson were neck and neck to the end, when Watson edged ahead in Final Jeopardy. Again, Rutter coasted to third place. Ferrucci said that he and his team left the practice rounds thinking, “Ken’s really good--but what’s going on with Brad?”

When Ferrucci pulled in to the Yorktown labs the morning of the match, the site had been transformed for the event. The visitors’ parking lot was cordoned off for VIPs. Security guards posted at the doors checked every person entering the building, matching their names against a list. And in the vast lobby, usually manned by one lonely guard, IBM’s luminaries and privileged guests circled around tables piled with brunch-fare. Ferrucci made his way to Watson’s old practice studio, now refashioned as an exhibition room. There he gave a half-hour talk about the supercomputer to a gathering of IBM clients, including J.P. Morgan, American Express, and the pharmaceutical giant Merck and Co. Ferrucci recalled the distant days when a far stupider Watson responded to a clue about a famous French bacteriologist by saying: “What is ‘How Tasty Was My Little Frenchman’?” (That was the title of a 1971 Brazilian comedy about cannibals in the Amazon.) 

His next stop, the make-up room, revealed his true state of mind. The make-up artist was a woman originally from Italy, like much of Ferrucci's family. As she began to work on his face she showered him with warmth and concern--acting "motherly." This rekindled his powerful feelings about his team and the end of their journey, and before he knew it, tears were streaming down his face. The more the woman comforted him, the worse it got. Ferrucci finally staunched the flow and got the pancake on his face. But he knew he was a mess. He hunted down Scott Brooks, the light-hearted press officer. Maybe some jokes, he thought, “would take the lump out of my throat.” Brooks laughed and kidded his colleague.

This irritated the testy Ferrucci and, to his relief, knocked him out of his fragile mood. He joined his team for one last lunch, all of them seated at a long table in the cafeteria. As they were finishing, just a few minutes before 1 p.m., a roaring engine interrupted conversations in the cafeteria. It was IBM’s Chairman Sam Palmisano landing in his helicopter. The hour had come. Ferrucci walked down the sunlit corridor to the auditorium.


****



Ken Jennings woke up that Friday morning in the Crown Plaza in White Plains. He’d slept well, much better than he usually did before big Jeopardy matches. Jennings had reason to feel confident. He had destroyed Watson in one of the practice rounds. Afterwards, he said, Watson’s developers told him that the game had featured a couple of “train wrecks”--categories where Watson appeared disoriented. Children’s literature was one. For Jennings, train wrecks signaled the machine’s vulnerability. With a few of them in the big match, he could stand up tall for humans, and perhaps extend his legend from Jeopardy to the broader realm of knowledge. “Given the right board,” he said, “Watson is beatable.” A stakes were considerable. While IBM would give all of Watson’s winnings to charity, a human winner would earn a half million-dollar prize, with another half million to give to the charity of his choice. Finishing in second or third place was worth $150,000 and $100,000, with equal amounts for the players’ charities.

A little after 11, a car service stopped by the hotel, picked up Jennings and his wife, Mindy, and drove them 13 miles north to IBM’s Yorktown laboratory. Jennings carried three changes of clothes, so that he could dress differently for each session, simulating three different days. As soon as he stepped out of the car, Jeopardy officials whisked him past the crush of people in the lobby and toward the staircase. Jeopardy had cleared out a couple of offices in IBM’s Human Resources department, and Jennings was given one as a dressing room.


On short visits to the East Coast, Brad Rutter liked to sleep late, so that he stayed in sync with West Coast time. But the morning of the match, he found himself awake at 7, which meant he faced four and a half hours before the car came by. Rutter was at the Ritz Carlton in White Plains, about a half mile from Jennings. He breakfasted, showered, and then killed time until 11:30. Unlike Jennings, Rutter had grounds for serious concern. In the practice rounds, he had been uncharacteristically slow. The computer had an exquisite sense of timing, and Jennings seemed to hold his own. Rutter, who had never lost a Jeopardy game in his life, was facing a flame-out unless he could get to the buzzer fast.

Shortly after Rutter arrived at IBM, he and Jennings played one last practice round with Watson. To Rutter’s delight, his buzzer thumb started to regain the old magic. He beat both Jennings and the machine. Now, in the three practice matches, each of the three players had registered a win. But Jennings and Rutter noticed something strange about Watson. Its game strategy, Jennings said, “seemed naive.” Just like beginning Jeopardy players, Watson started with the easy low-dollar clues and moved straight down the board. Why wasn’t it hunting for Daily Doubles? In the Blue-ray disks given to them in November, Jennings and Rutter had seen that Watson skipped around the high-dollar clues, hunting for the single Daily Double on the first Jeopardy board, and the two in Double Jeopardy. Landing Daily Doubles was vital. It gave a player the means to build a big lead. Equally important, once Daily Doubles were off the board, the leader was hard to catch. But in the practice rounds, Watson didn’t appear to have this strategy in mind.

The two players were led to a tiny entry hall behind the auditorium. As the event commenced, shortly after one p.m., they waited. They listened as IBM introduced Watson to its customers. “You know how they call time outs before a guy kicks a field goal?” Jennings said. “We were joking that they were doing the same thing to us. Icing us.” Through the door they heard speeches by John Kelly, the chief of IBM Research, and Sam Palmisano. Harry Friedman, who decades earlier had earned $5 a joke as a writer for Hollywood Squares, delivered one of his own. “I’ve lived in Hollywood for a long time,” he told the crowd. “So I know something about Artificial Intelligence.” When Ferrucci was called on to the stage, the crowd rose for a standing ovation. “I already cried in make-up,” he said. “Let’s not repeat that.”



Finally, it was time for Jeopardy. Jennings and Rutter were summoned to the stage. They walked down the narrow aisle of the auditorium, Jennings leading in a business suit and yellow tie, the taller loose-gaited Rutter following him, his collar unbuttoned. They settled at their lecterns, Jennings on the far side, Rutter closer to the crowd. Between them, its circular black screen dancing with jagged colorful lines, sat Watson.

The show began with its familiar music. A fill-in for legendary announcer, Johnny Gilbert (who hadn’t made the trip from Culver City), introduced the contestants and Alex Trebek. But even then, Jennings and Rutter had to wait while an IBM video told the story of the Watson project. In a second video, Trebek talked to Ferrucci about the machinery behind the bionic player--now up to 2,880 processing cores. Then Trebek gave viewers a tutorial on Watson’s answer panel. This would reveal the statistical confidence that the computer had in each of its top responses. It was a window into Watson’s thinking.

Trebek, in fact, had been a late convert to the answer panel. Like the rest of the Jeopardy team, he was loath to stray from the show’s time-honored formulas. People knew what to expect from the game: the precise movements of the cameras, the familiar music, voices and categories. Wouldn’t the intrusion of an electronic answer panel distract them, and ultimately make the game less enjoyable to watch? He raised that concern on a visit to IBM in November. But the prospect of playing the game without Watson’s answer panel horrified Ferrucci. Millions of viewers, he believed, would simply conclude that the machine had been fed all the answers. They wouldn’t appreciate what Watson had gone through to arrive at the correct response. So while Trebek was eating lunch that day, Ferrucci carried out an experiment. He had his technicians take down the answer panel. When the afternoon sessions began, it only took one game for Trebek to ask for the answer panel back.  Later, he said, watching Watson without its analysis was “boring as hell.”


A hush settled over the auditorium. Finally, it was time to play. Ferrucci, sitting between David Gondek and Eric Brown, laced his hands tightly and made a steeple with his index fingers. He watched as Trebek, with a wave of his arm, revealed the six categories for the first round of Jeopardy....