Stephen Baker

The Boost
Home - Viewing one post

Final Jeopardy: How can Watson conclude that Toronto is a U.S. city?

February 15, 2011Jeopardy book


When I met yesterday with IBM's chief scientist behind Jeopardy, David Ferrucci, he was wearing a Toronto Blue Jays jacket. It had to do with Watson's only signficant blooper in an otherwise dominant performance in the second half of its first game.

The Final Jeopardy category was U.S. Cities. The clue: "Its largest airport is named for a World War II hero, its second largest for a World War II battle." Watson, strangely, came up with the response: "What is Toronto??????" It was programmed to add all those question marks to show the audience that it had very low confidence in the response. But still, how could it choose Toronto in a category for U.S. cities.

After the game, Ferrucci and his team were eager to explain Watson's thinking process. Strangely, from a PR point of view, they seemed determined to focus on one moment of weakness in session that exhibited Watson's strengths. But they have poured four years of research into this machine, and they like to clear up doubts.

A few key issues:

1)  Watson can never be sure of anything. Is it possible that the old rock star Alice Cooper is a man? If Watson finds enough evidence, it will bet on it--even though the name "Alice" is sure to create a lot of doubt. This flexibility in its thinking can save Watson from gaffes--but also lead to a few.

2) Category titles cannot be trusted. I blogged about this earlier, in a post How Watson Thinks. It has learned through exhaustive statistical analysis that many clues do not jibe with categories. A category about US novelists, for example, can ask about J.D. Salinger's masterpiece. Catcher in the Rye is a novel, not a novelist! These things happen time and again, and Watson notices. So it pays scant attention to the categories.

3) If this had been a normal Jeopardy clue, Watson would not have buzzed. It had only 14% confidence in Toronto (whose Pearson airport is named for a man who was active in World War two One), and 11% in Chicago. Watson simply did not come up with the answer, and Toronto was its guess.

Even so, how could it guess that Toronto was an American city? Here we come to the weakness of statistical analysis. While searching through data, it notices that the United States is often called America. Toronto is a North American city. Its baseball team, the Blue Jays, plays in the American League. (That's why Ferrucci was wearing a Blue Jay jacket). If Watson happened to study the itinerary of my The Numerati book tour, it included a host of American cities, from Philadelphia and Pittsburgh, to Seattle, San Francisco, and Toronto. In documents like that, people often don't stop to note for inquiring computers that Toronto actually shouldn't be placed in the group.

Long story short: Watson blew it. It happens. It earned it lots of abuse on Twitter, such as:

Watson the computer is a dumbass. Doesn't even know that Toronto is not in the US. ...

But now, hopefully, Lutheranish and other critics will understand why.

add comment share:






©2014 Stephen Baker Media, All rights reserved.     Site by Infinet Design







Kirkus Review - Kirkus

Publishers' Weekly - Publishers' Weekly

Booklist - Booklist

Kirkus - Kirkus Reviews

read more reviews



The Boost: an excerpt
- April 15, 2014


My horrible Superbowl weekend, in perspective
- February 3, 2014


My coming novel: Boosting human cognition
- May 30, 2013


Why Nate Silver is never wrong
- November 8, 2012


The psychology behind bankers' hatred for Obama
- September 10, 2012


"Corporations are People": an op-ed
- August 16, 2011


Wall Street Journal excerpt: Final Jeopardy
- February 4, 2011


Why IBM's Watson is Smarter than Google
- January 9, 2011


Rethinking books
- October 3, 2010


The coming privacy boom
- August 17, 2010


The appeal of virtual
- May 18, 2010


My next book: IBM's Jeopardy mission
- March 22, 2010