Netflix Prize: Numerati go mainstream - Stephen Baker

Home - Viewing one post

Netflix Prize: Numerati go mainstream posted on September 21, 2009

Datamining

What should Netflix computers conclude when a sci-fi lover starts checking out When Harry Met Sally?

I was in the San Rafael (Calif) offices of Fair Isaac nearly three years ago, interviewing the ~~financial quants~~ statisticians and dataminers there for the Numerati. Suddenly, there was a commotion. News had just arrived that Netflix would be sharing loads of anonymous behavioral data, and offering $1 million to anyone who could improve the accuracy of their movie recommendations by 10%.

People at Fair Isaac were excited. They didn't care about the $1 million. They just wanted to get their hands on all that data. It seemed to me that I had a window into a fascinating niche of the Numerati world.

Fast forward to this morning. I walk over the the sumptuous Four Season's Hotel, on E. 57th Street, where loads of journalists, including the Times and the Journal and AP, are squeezed into a ballroom for the announcement of the Netflix winners. My read: What seemed like a niche three years ago, the crunching of our behavioral data--the mathematical modeling of humanity--is now mainstream. They also announced that a second Netflix prize will soon be announced. Here's my account on BusinessWeek.com.

I called up Darren Vengroff. He's the chief scientist at RichRelevance, a recommendation start-up I've written about before. He told me that the upcoming Netflix Prize II, which will include much more demographic detail, could lead to much more sophisticated analysis. For example, if the service knows that a certain user loves sci-fi and hates romantic movies, what should it do if one day he starts checking out When Harry Met Sally and Sleepless in Seattle? It might conclude, Vengroff suggests, that someone else is at his computer. Or perhaps he's preparing for company. By engaging dataminers around the globe, Netflix may soon be able to draw such conclusions.