Home - Viewing one post
Final Jeopardy: How can Watson conclude that Toronto is a U.S. city?
|When I met yesterday with IBM's chief scientist behind Jeopardy, David Ferrucci, he was wearing a Toronto Blue Jays jacket. It had to do with Watson's only signficant blooper in an otherwise dominant performance in the second half of its first game.
The Final Jeopardy category was U.S. Cities. The clue: "Its largest airport is named for a World War II hero, its second largest for a World War II battle." Watson, strangely, came up with the response: "What is Toronto??????" It was programmed to add all those question marks to show the audience that it had very low confidence in the response. But still, how could it choose Toronto in a category for U.S. cities.
After the game, Ferrucci and his team were eager to explain Watson's thinking process. Strangely, from a PR point of view, they seemed determined to focus on one moment of weakness in session that exhibited Watson's strengths. But they have poured four years of research into this machine, and they like to clear up doubts.
A few key issues:
1) Watson can never be sure of anything. Is it possible that the old rock star Alice Cooper is a man? If Watson finds enough evidence, it will bet on it--even though the name "Alice" is sure to create a lot of doubt. This flexibility in its thinking can save Watson from gaffes--but also lead to a few.
2) Category titles cannot be trusted. I blogged about this earlier, in a post How Watson Thinks. It has learned through exhaustive statistical analysis that many clues do not jibe with categories. A category about US novelists, for example, can ask about J.D. Salinger's masterpiece. Catcher in the Rye is a novel, not a novelist! These things happen time and again, and Watson notices. So it pays scant attention to the categories.
3) If this had been a normal Jeopardy clue, Watson would not have buzzed. It had only 14% confidence in Toronto (whose Pearson airport is named for a man who was active in World War
two One), and 11% in Chicago. Watson simply did not come up with the answer, and Toronto was its guess.
Even so, how could it guess that Toronto was an American city? Here we come to the weakness of statistical analysis. While searching through data, it notices that the United States is often called America. Toronto is a North American city. Its baseball team, the Blue Jays, plays in the American League. (That's why Ferrucci was wearing a Blue Jay jacket). If Watson happened to study the itinerary of my The Numerati book tour, it included a host of American cities, from Philadelphia and Pittsburgh, to Seattle, San Francisco, and Toronto. In documents like that, people often don't stop to note for inquiring computers that Toronto actually shouldn't be placed in the group.
Long story short: Watson blew it. It happens. It earned it lots of abuse on Twitter, such as:
But now, hopefully, Lutheranish and other critics will understand why.
RT @sbauman: Report Uncovers Huge Business Opportunities in Healthcare Data Analytics
follow me on twitter
Kirkus - Kirkus Reviews
Andrew Dunn - Bloomberg News
Culture Mob - Dan Sampson
Shelfari (Amazon) - Tom Nissley
read more reviews
Why Nate Silver is never wrong
- November 8, 2012
The psychology behind bankers' hatred for Obama
- September 10, 2012
"Corporations are People": an op-ed
- August 16, 2011
Wall Street Journal excerpt: Final Jeopardy
- February 4, 2011
Why IBM's Watson is Smarter than Google
- January 9, 2011
- October 3, 2010
The coming privacy boom
- August 17, 2010
The appeal of virtual
- May 18, 2010
My next book: IBM's Jeopardy mission
- March 22, 2010
- November 12, 2009
BusinessWeek cannot afford to stay within McGraw-Hill
- August 6, 2009
How to remake BusinessWeek?
- July 16, 2009