When we started collecting food reviews through our app we faced the decision of what scale we should use. Was a 10-point scale or a 5-star rating the best option? The arguments for which scale works best in the world of ratings and research is long, drawn-out and largely inconclusive. After a our own internal debates and testing we discovered that we should use the most instinctive scale possible - a thumbs up and down rating system.
Netflix have just announced that they are moving from a 5-star rating system to a thumbs up and down system for some of the key reasons we chose the same method:
Netflix VP of Product Todd Yellin told journalists on Thursday, that the company had tested the new thumbs up and down ratings with hundred of thousands of members in 2016. “We are addicted to the methodology of A/B testing,” Yellin said. The result was that thumbs got 200% more ratings than the traditional star-rating feature.
Volume of ratings matter. Currently most restaurants will only receive a few reviews or ratings a week on sites like TripAdvisor. These limited reviews are not actionable and often misleading due of the law of small numbers - a single bad review can significantly skew all the results. Our emphasis on volume means that we currently collect as many reviews in just 1.2 days that TripAdvisor collects in a whole year.
Over time, Netflix realized that explicit star ratings were less relevant than other signals. Users would rate documentaries with 5 stars, and comedy movies with just 3 stars, but still watch comedy movies more often than those high-rated documentaries.
It's the same with food. Switch documentaries for fine dining restaurants (or high concept cuisine) and comedy movies for casual dining restaurants (or comfort foods) and you'll see the same effect. If you really want to judge if a dish is good or bad you need to take out preconceived expectations. To see what people instinctively feel about restaurants and food you just have to ask wether they'd give the dish a thumbs up or down.
“We made ratings less important because the implicit signal of your behavior is more important,” Yellin said .
The reason Netflix is making the change is to improve their ability to provide personalised recommendations to their customers. From a large collection of apparently simple ratings, incredible insights can be produced.
Here's the full article from Variety