The Ratings We Gave

Netflix’s 2006 prize made recommendation engineering into a public challenge. In an October 6 interview with MIT Technology Review’s Kate Greene, Jim Bennett, Netflix’s vice president of recommendation systems, explained the bargain: outside researchers would get a large ratings dataset and a chance to improve on Cinematch, the company’s recommendation system. Netflix would get ideas its own team could not test exhaustively.

That is a useful opening, not merely a marketing stunt. Bennett identified access to substantial datasets as a constraint on research. Releasing one gave outsiders something concrete to work with. An invitation to innovate means rather more when it includes the material needed to run the experiment.

But look at what made that material valuable.

Bennett described a system that compared ratings across pairs of movies, looking for relationships between what people liked and disliked. Sparse ratings made obscure titles harder to handle. People also used the star scale differently: some favored the extremes; others gave everything three stars. The engineering challenge was not simply collecting opinions. It was finding useful relationships among inconsistent expressions of taste.

The supposedly personal recommendation was therefore a collective construction. Other people’s judgments helped make a prediction about yours. The customer looking at a few colored stars could easily miss the crowd holding them up.

The competition turned that relationship into a measurable task. Bennett said Netflix released 100 million ratings while withholding another three million against which predictions could be checked. Greene reported a $1 million prize for a ten-percent improvement over Cinematch.

That test provides discipline: an entrant cannot establish superiority merely by declaring its algorithm revolutionary. Somewhere, mercifully, there is an answer to compare.

It also draws a boundary around what success means. Predicting a withheld rating more accurately is not the same as establishing that a customer has more control over recommendations, or that the people contributing ratings have a meaningful say in their reuse. Those are different questions. A benchmark cannot answer them by getting smaller.

Nor does the available reporting justify calling the arrangement simple theft. Customers could receive better recommendations; researchers gained access to useful material. Collective contribution does not, by itself, settle legal ownership or prove that every commercial benefit is illegitimate. The important issue is whether the return is adequate and whether contributors can understand the bargain.

A coherenceist test keeps those contributors in view after their judgments become data. What uses were explained to them? Could they challenge those uses? What benefit reached them beyond the promise that the service would improve? Greene’s interview explains the technical exchange with researchers; it does not establish the answers to those customer-governance questions.

Opening a research problem and sharing authority over its inputs are separate achievements. Netflix’s invitation deserves credit for the first without being awarded the second by implication.

The prize put a price on improving the prediction. The people whose judgments made prediction possible remained much harder to see in the accounting. Apparently the algorithm was not the only thing that needed better resolution.

Historical dating note

This retrospective retains the assigned September 21, 2006 archive date, but that date is not verified as the prize’s launch. The supplied Netflix announcement URL was blocked during retrieval. Greene’s October 6 report says the announcement occurred earlier that week; this article relies on that retrieved reporting, not on an assumed September announcement.