Netflix Prize

Last updated

The Netflix Prize was an open competition for the best collaborative filtering algorithm to predict user ratings for films, based on previous ratings without any other information about the users or films, i.e. without the users being identified except by numbers assigned for the contest.

Contents

The competition was held by Netflix, a video streaming service, and was open to anyone who is neither connected with Netflix (current and former employees, agents, close relatives of Netflix employees, etc.) nor a resident of certain blocked countries (such as Cuba or North Korea). [1] On September 21, 2009, the grand prize of US$1,000,000 was given to the BellKor's Pragmatic Chaos team which bested Netflix's own algorithm for predicting ratings by 10.06%. [2]

Problem and data sets

Netflix provided a training data set of 100,480,507 ratings that 480,189 users gave to 17,770 movies. Each training rating is a quadruplet of the form <user, movie, date of grade, grade>. The user and movie fields are integer IDs, while grades are from 1 to 5 (integer) stars. [3]

The qualifying data set contains over 2,817,131 triplets of the form <user, movie, date of grade>, with grades known only to the jury. A participating team's algorithm must predict grades on the entire qualifying set, but they are informed of the score for only half of the data: a quiz set of 1,408,342 ratings. The other half is the test set of 1,408,789, and performance on this is used by the jury to determine potential prize winners. Only the judges know which ratings are in the quiz set, and which are in the test set—this arrangement is intended to make it difficult to hill climb on the test set. Submitted predictions are scored against the true grades in the form of root mean squared error (RMSE), and the goal is to reduce this error as much as possible. Note that, while the actual grades are integers in the range 1 to 5, submitted predictions need not be. Netflix also identified a probe subset of 1,408,395 ratings within the training data set. The probe, quiz, and test data sets were chosen to have similar statistical properties.

In summary, the data used in the Netflix Prize looks as follows:

For each movie, the title and year of release are provided in a separate dataset. No information at all is provided about users. In order to protect the privacy of the customers, "some of the rating data for some customers in the training and qualifying sets have been deliberately perturbed in one or more of the following ways: deleting ratings; inserting alternative ratings and dates; and modifying rating dates." [2]

The training set is constructed such that the average user rated over 200 movies, and the average movie was rated by over 5000 users. But there is wide variance in the data—some movies in the training set have as few as 3 ratings, [4] while one user rated over 17,000 movies. [5]

There was some controversy as to the choice of RMSE as the defining metric. It has been claimed that even as small an improvement as 1% RMSE results in a significant difference in the ranking of the "top-10" most recommended movies for a user. [6]

Prizes

Prizes were based on improvement over Netflix's own algorithm, called Cinematch, or the previous year's score if a team has made improvement beyond a certain threshold. A trivial algorithm that predicts for each movie in the quiz set its average grade from the training data produces an RMSE of 1.0540. Cinematch uses "straightforward statistical linear models with a lot of data conditioning." [7]

Using only the training data, Cinematch scores an RMSE of 0.9514 on the quiz data, roughly a 10% improvement over the trivial algorithm. Cinematch has a similar performance on the test set, 0.9525. In order to win the grand prize of $1,000,000, a participating team had to improve this by another 10%, to achieve 0.8572 on the test set. [2] Such an improvement on the quiz set corresponds to an RMSE of 0.8563.

As long as no team won the grand prize, a progress prize of $50,000 was awarded every year for the best result thus far. However, in order to win this prize, an algorithm had to improve the RMSE on the quiz set by at least 1% over the previous progress prize winner (or over Cinematch, the first year). If no submission succeeded, the progress prize was not to be awarded for that year.

To win a progress or grand prize a participant had to provide source code and a description of the algorithm to the jury within one week after being contacted by them. Following verification the winner also had to provide a non-exclusive license to Netflix. Netflix would publish only the description, not the source code, of the system. (To keep their algorithm and source code secret, a team could choose not to claim a prize.) The jury also kept their predictions secret from other participants. A team could send as many attempts to predict grades as they wish. Originally submissions were limited to once a week, but the interval was quickly modified to once a day. A team's best submission so far counted as their current submission.

Once one of the teams succeeded to improve the RMSE by 10% or more, the jury would issue a last call, giving all teams 30 days to send their submissions. Only then, the team with best submission was asked for the algorithm description, source code, and non-exclusive license, and, after successful verification; declared a grand prize winner.

The contest would last until the grand prize winner was declared. Had no one received the grand prize, it would have lasted for at least five years (until October 2, 2011). After that date, the contest could have been terminated at any time at Netflix's sole discretion.

Progress over the years

The competition began on October 2, 2006. By October 8, a team called WXYZConsulting had already beaten Cinematch's results. [8]

By October 15, there were three teams who had beaten Cinematch, one of them by 1.06%, enough to qualify for the annual progress prize. [9] By June 2007 over 20,000 teams had registered for the competition from over 150 countries. 2,000 teams had submitted over 13,000 prediction sets. [3]

Over the first year of the competition, a handful of front-runners traded first place. The more prominent ones were: [10]

On August 12, 2007, many contestants gathered at the KDD Cup and Workshop 2007, held at San Jose, California. [11] During the workshop all four of the top teams on the leaderboard at that time presented their techniques. The team from IBM Research—Yan Liu, Saharon Rosset, Claudia Perlich, and Zhenzhen Kou—won the third place in Task 1 and first place in Task 2.

Over the second year of the competition, only three teams reached the leading position:

2007 Progress Prize

On September 2, 2007, the competition entered the "last call" period for the 2007 Progress Prize. Over 40,000 teams from 186 countries had entered the contest. They had thirty days to tender submissions for consideration. At the beginning of this period the leading team was BellKor, with an RMSE of 0.8728 (8.26% improvement), followed by Dinosaur Planet (RMSE = 0.8769; 7.83% improvement), [12] and Gravity (RMSE = 0.8785; 7.66% improvement). In the last hour of the last call period, an entry by "KorBell" took first place. This turned out to be an alternate name for Team BellKor. [13]

On November 13, 2007, team KorBell (formerly BellKor) was declared the winner of the $50,000 Progress Prize with an RMSE of 0.8712 (8.43% improvement). [14] The team consisted of three researchers from AT&T Labs, Yehuda Koren, Robert Bell, and Chris Volinsky. [15] As required, they published a description of their algorithm. [16]

2008 Progress Prize

The 2008 Progress Prize was awarded to the team BellKor. Their submission combined with a different team, BigChaos achieved an RMSE of 0.8616 with 207 predictor sets. [17] The joint-team consisted of two researchers from Commendo Research & Consulting GmbH, Andreas Töscher and Michael Jahrer (originally team BigChaos) and three researchers from AT&T Labs, Yehuda Koren, Robert Bell, and Chris Volinsky (originally team BellKor). [18] As required, they published a description of their algorithm. [19] [20]

This was the final Progress Prize because obtaining the required 1% improvement over the 2008 Progress Prize would be sufficient to qualify for the Grand Prize. The prize money was donated to the charities chosen by the winners.

2009

On July 25, 2009 the team "The Ensemble", a merger of the teams "Grand Prize Team" and "Opera Solutions and Vandelay United," achieved a 10.09% improvement over Cinematch (a Quiz RMSE of 0.8554). [21] [22]

On June 26, 2009 the team "BellKor's Pragmatic Chaos", a merger of teams "Bellkor in BigChaos" and "Pragmatic Theory", achieved a 10.05% improvement over Cinematch (a Quiz RMSE of 0.8558). The Netflix Prize competition then entered the "last call" period for the Grand Prize. In accord with the Rules, teams had thirty days, until July 26, 2009 18:42:37 UTC, to make submissions that will be considered for this Prize. [23]

On July 26, 2009, Netflix stopped gathering submissions for the Netflix Prize contest. [24]

The final standing of the Leaderboard at that time showed that two teams met the minimum requirements for the Grand Prize. "The Ensemble" with a 10.10% improvement over Cinematch on the Qualifying set (a Quiz RMSE of 0.8553), and "BellKor's Pragmatic Chaos" with a 10.09% improvement over Cinematch on the Qualifying set (a Quiz RMSE of 0.8554). [25] [26] The Grand Prize winner was to be the one with the better performance on the Test set.

On September 18, 2009, Netflix announced team "BellKor's Pragmatic Chaos" as the prize winner (a Test RMSE of 0.8567), and the prize was awarded to the team in a ceremony on September 21, 2009. [27] "The Ensemble" team had matched BellKor's result, but since BellKor submitted their results 20 minutes earlier, the rules award the prize to BellKor. [22] [28]

The joint-team "BellKor's Pragmatic Chaos" consisted of two Austrian researchers from Commendo Research & Consulting GmbH, Andreas Töscher and Michael Jahrer (originally team BigChaos), two researchers from AT&T Labs, Robert Bell, and Chris Volinsky, Yehuda Koren from Yahoo! (originally team BellKor) and two researchers from Pragmatic Theory, Martin Piotte and Martin Chabbert. [29] As required, they published a description of their algorithm. [30]

The team reported to have achieved the "dubious honors" (sic Netflix) of the worst RMSEs on the Quiz and Test data sets from among the 44,014 submissions made by 5,169 teams was "Lanterne Rouge", led by J.M. Linacre, who was also a member of "The Ensemble" team.

Cancelled sequel

On March 12, 2010, Netflix announced that it would not pursue a second Prize competition that it had announced the previous August. The decision was in response to a lawsuit and Federal Trade Commission privacy concerns. [31]

Privacy concerns

Although the data sets were constructed to preserve customer privacy, the Prize has been criticized by privacy advocates. In 2007 two researchers from The University of Texas at Austin were able to identify individual users by matching the data sets with film ratings on the Internet Movie Database. [32] [33]

On December 17, 2009, four Netflix users filed a class action lawsuit against Netflix, alleging that Netflix had violated U.S. fair trade laws and the Video Privacy Protection Act by releasing the datasets. [34] There was public debate about privacy for research participants. On March 19, 2010, Netflix reached a settlement with the plaintiffs, after which they voluntarily dismissed the lawsuit.

See also

Related Research Articles

Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of statistical algorithms that can learn from data and generalize to unseen data, and thus perform tasks without explicit instructions. Recently, generative artificial neural networks have been able to surpass many previous approaches in performance.

<span class="mw-page-title-main">Collaborative filtering</span> Algorithm

Collaborative filtering (CF) is a technique used by recommender systems. Collaborative filtering has two senses, a narrow one and a more general one.

Topcoder is a crowdsourcing company with an open global community of designers, developers, data scientists, and competitive programmers. Topcoder pays community members for their work on the projects and sells community services to corporate, mid-size, and small-business clients. Topcoder also organizes the annual Topcoder Open tournament and a series of smaller regional events.

A recommender system, or a recommendation system, is a subclass of information filtering system that provides suggestions for items that are most pertinent to a particular user. Recommender systems are particularly useful when an individual needs to choose an item from a potentially overwhelming number of items that a service may offer.

PAQ is a series of lossless data compression archivers that have gone through collaborative development to top rankings on several benchmarks measuring compression ratio. Specialized versions of PAQ have won the Hutter Prize and the Calgary Challenge. PAQ is free software distributed under the GNU General Public License.

<span class="mw-page-title-main">AT&T Labs</span> Research and development division of AT&T

AT&T Labs, Inc. is the research & development division of AT&T, the telecommunications company. It employs some 1,800 people in various locations, including: Bedminster NJ; Middletown, NJ; Manhattan, NY; Warrenville, IL; Austin, TX; Dallas, TX; Atlanta, GA; San Francisco, CA; San Ramon, CA; and Redmond, WA. The main research division, made up of around 450 people, is based across the Bedminster, Middletown, San Francisco, and Manhattan locations.

Google Personalized Search is a personalized search feature of Google Search, introduced in 2004. All searches on Google Search are associated with a browser cookie record. When a user performs a search, the search results are not only based on the relevance of each web page to the search term, but also on which websites the user visited through previous search results. This provides a more personalized experience that can increase the relevance of the search results for the particular user. Such filtering may also have side effects, such as the creation of a filter bubble.

The Gordon Bell Prize is an award presented by the Association for Computing Machinery each year in conjunction with the SC Conference series. The prize recognizes outstanding achievement in high-performance computing applications. The main purpose is to track the progress over time of parallel computing, by acknowledging and rewarding innovation in applying high-performance computing to applications in science, engineering, and large-scale data analytics. The prize was established in 1987. A cash award of $10,000 accompanies the recognition, funded by Gordon Bell, a pioneer in high-performance and parallel computing.

The Hutter Prize is a cash prize funded by Marcus Hutter which rewards data compression improvements on a specific 1 GB English text file, with the goal of encouraging research in artificial intelligence (AI).

Cold start is a potential problem in computer-based information systems which involves a degree of automated data modelling. Specifically, it concerns the issue that the system cannot draw any inferences for users or items about which it has not yet gathered sufficient information.

GARUDA(Global Access to Resource Using Distributed Architecture) is India's Grid Computing initiative connecting 17 cities across the country. The 45 participating institutes in this nationwide project include all the IITs and C-DAC centers and other major institutes in India.

There are a number of competitions and prizes to promote research in artificial intelligence.

<span class="mw-page-title-main">GroupLens Research</span> Computer science research lab

GroupLens Research is a human–computer interaction research lab in the Department of Computer Science and Engineering at the University of Minnesota, Twin Cities specializing in recommender systems and online communities. GroupLens also works with mobile and ubiquitous technologies, digital libraries, and local geographic information systems.

<span class="mw-page-title-main">Pixiv</span> Japanese online artist community

Pixiv is a Japanese online community for artists. It was first launched as a beta test on September 10, 2007, by Takahiro Kamitani and Takanori Katagiri. Pixiv Inc. is headquartered in Sendagaya, Shibuya, Tokyo, Japan. As of January 2024, the site consists of over 100 million members, over 115 million illustration and manga and over 21 million novel work posted. Pixiv aims to provide a place for artists to exhibit their illustrations and get feedback via a rating system and user comments. Works are organized in an extensive tag structure which forms the backbone of the website.

An innovation competition is a method or process of the industrial process, product or business development. It is a form of social engineering, which focuses to the creation and elaboration of the best and sustainable ideas, coming from the best innovators.

<span class="mw-page-title-main">Kaggle</span> Internet platform for data science competitions

Kaggle is a data science competition platform and online community of data scientists and machine learning practitioners under Google LLC. Kaggle enables users to find and publish datasets, explore and build models in a web-based data science environment, work with other data scientists and machine learning engineers, and enter competitions to solve data science challenges.

Gravity R&D is an IT vendor specialized in recommender systems. Gravity was founded by members of the Netflix Prize team "Gravity".

Netflix is a subscription streaming service owned by the American company Netflix, Inc. Launched on August 29, 1997, it initially offered DVD rental and sale by mail, but the sales were eliminated within a year to focus on the DVD rental business. In 2007, the company began transitioning to its current subscription streaming model. Since its launch, the company was subject to numerous criticisms, the basis of which range from its business practices and workplace culture to issues with the service it provides, including content issues, lack of close captioning and pricing. This article provides an overview of key criticisms the company has faced.

Matrix factorization is a class of collaborative filtering algorithms used in recommender systems. Matrix factorization algorithms work by decomposing the user-item interaction matrix into the product of two lower dimensionality rectangular matrices. This family of methods became widely known during the Netflix prize challenge due to its effectiveness as reported by Simon Funk in his 2006 blog post, where he shared his findings with the research community. The prediction results can be improved by assigning different regularization weights to the latent factors based on items' popularity and users' activeness.

ACM Conference on Recommender Systems is a peer-reviewed academic conference series about recommender systems. Sponsored by the Association for Computing Machinery. This conference series focuses on issues such as algorithms, machine learning, human-computer interaction, and data science from a multi-disciplinary perspective. The conference community includes computer scientists, statisticians, social scientists, psychologists, and others.

References

  1. "The Netflix Prize Rules" (PDF). Archived from the original (PDF) on 2020-05-10. Retrieved 2019-11-06.
  2. 1 2 3 "The Netflix Prize". Archived from the original on 2009-09-24. Retrieved 2012-07-09.
  3. 1 2 James Bennett; Stan Lanning (August 12, 2007). "The Netflix Prize" (PDF). Proceedings of KDD Cup and Workshop 2007. Archived from the original (PDF) on September 27, 2007. Retrieved 2007-08-25.
  4. Sigmoid Curve (2006-10-08). "Miss Congeniality". Netflix Prize Forum. Archived from the original on 2012-02-06. Retrieved 2007-08-25.
  5. prodigious (2006-10-06). "A single customer that rated 17,000 movies". Netflix Prize Forum. Archived from the original on 2012-02-06. Retrieved 2007-08-25.
  6. YehudaKoren (2007-12-18). "How useful is a lower RMSE?". Netflix Prize Forum. Archived from the original on 2012-02-06.
  7. "Netflix Prize Frequently Asked Questions". Archived from the original on 2007-08-21. Retrieved 2007-08-21.
  8. "Netflix Prize Rankings". Hacking NetFlix. October 9, 2006. Archived from the original on 2006-10-30. Retrieved 2007-08-21.
  9. "Netflix Prize (I tried to resist, but...)". Juho Snellman's Weblog. October 15, 2006. Retrieved 2007-08-21.
  10. "Top contenders for Progress Prize 2007 chart".
  11. "The KDD Cup and Workshop 2007".
  12. "Dinosaur Planet". 2022-12-08.
  13. admin (2022-08-28). "BellKor's Pragmatic Chaos Wins $1 Million Netflix Prize by Mere Minutes". Populousness. Retrieved 2022-08-28.
  14. Prizemaster (2007-11-13). "Netflix Progress Prize 2007 awarded to team KorBell". Netflix Prize Forum. Archived from the original on 2012-02-06.
  15. "$50,000 Progress Prize is Awarded on First Anniversary of $1 Million Netflix Prize". Netflix .
  16. R. Bell; Y. Koren; C. Volinsky (2007). "The BellKor solution to the Netflix Prize". CiteSeerX   10.1.1.142.9009 .
  17. Robert Bell; Yehuda Koren; Chris Volinsky (2008-12-10). "The BellKor 2008 Solution to the Netflix Prize" (PDF). Netflix Prize Forum.
  18. "Netflix Awards $50,000 Progress Prize in Year Two of Multi-Year, Multi-National Netflix Prize Competition". Archived from the original on 2009-06-30. Retrieved 2009-06-22.
  19. A. Töscher; M. Jahrer (2008). "The BigChaos solution to the Netflix Prize 2008" (PDF). Archived from the original (PDF) on 2012-02-16. Retrieved 2009-06-24.
  20. R. Bell; Y. Koren; C. Volinsky (2008). "The BellKor solution to the Netflix Prize 2008" (PDF). Archived from the original (PDF) on 2012-02-16. Retrieved 2009-06-24.
  21. "The Ensemble". 2022-12-08. Archived from the original on 2014-12-27.
  22. 1 2 "Netflix Prize Leaderboard". 2009-07-26. Archived from the original on 2013-12-13. Retrieved 2013-12-09.
  23. "BellKor's Pragmatic Chaos". 2009-06-26.
  24. "Contest Closed". 2009-07-26. Archived from the original on 2009-07-28. Retrieved 2009-07-27.
  25. Lester Mackey (2022-12-08). "Final Submission Countdown". Archived from the original on 2014-12-27.
  26. "The Netflix Prize Comes To A Buzzer-Beater, Nailbiting Finish". 2009-07-26.
  27. "Grand Prize awarded to team BellKor's Pragmatic Chaos". Netflix Prize Forum. 2009-09-21. Archived from the original on 2012-05-07.
  28. Steve Lohr (2009-09-21). "A $1 Million Research Bargain for Netflix, and Maybe a Model for Others". New York Times.
  29. "Netflix Awards $1 Million Netflix Prize and Announces Second $1 Million Challenge". Archived from the original on 2009-09-25. Retrieved 2009-09-24.
  30. Andreas Töscher & Michael Jahrer (2009-09-05). "The BigChaos Solution to the Netflix Grand Prize" (PDF). commendo research & consulting. Retrieved 2022-11-02.
  31. "Netflix Prize Update". Netflix Prize Forum. 2010-03-12.
  32. Narayanan, Arvind; Shmatikov, Vitaly (2006). "How To Break Anonymity of the Netflix Prize Dataset". arXiv: cs/0610105 .
  33. Demerjian, Dave (15 March 2007). "Rise of the Netflix Hackers". wired.com. Wired. Retrieved 13 December 2014.
  34. Singel, Ryan. "Netflix Spilled Your Brokeback Mountain Secret, Lawsuit Claims". Wired. Retrieved 11 August 2017.