ReviewMeta Analysis Test: Skewed HelpfulnessApril 27th, 2016
A common feature on modern websites with a reviewing platform is the helpfulness score. This feature asks readers about the quality of a review, usually by asking the simple yes or no question “Was this review helpful to you?” Different websites use this score differently. Some websites just show the score for the benefit of readers. Other websites use the helpfulness score to determine the order of reviews on a given product page, placing the most helpful reviews in the most visible area of the page. Most dramatically, some sites, including Amazon, take the helpfulness of reviews into their calculation of the average rating for the product.
Despite being a tool for increasing review quality, unscrupulous brands or overzealous fans will use the helpfulness score to manipulate online reviews by marking negative reviews for their product as unhelpful, and positive reviews as helpful. Whether they are trying to maximize profit or are just angered at their favorite product’s negative reviews, readers will intentionally treat the helpfulness score as an agree or disagree button in order to promote the brand nearest to their heart (or their wallet). This manipulation can push the positive reviews to the top of a product page and negative reviews to the bottom of a product page, and it may even may cause the overall product rating to adjust in their favor. This form of manipulation is quick, easy and one of the sneakiest forms of review manipulation that one can conduct. It is very hard for even the most skeptical reader to detect since the helpfulness scores are only shown on individual reviews and the effects of the manipulation are largely hidden under the hood.
Although it is very difficult for individual readers to see helpfulness score manipulation, it is quite easy for ReviewMeta to detect. To do this we simply group reviews into two categories, negative reviews and positive reviews. Then we calculate the average net helpfulness score for both groups. The net helpfulness score is the number of times a review is marked as being helpful minus the number of times a review is marked as being not helpful. We then look at the average net helpfulness score for both negative reviews and positive reviews and compare the difference. If there is no helpfulness score manipulation, there should be very little difference between the net helpfulness scores for negative and positive reviews. If a difference is detected, we’ll check to see if this discrepancy is statistically significant. This means that we run the data through an equation that takes into account the total number of reviews along with the variance of the individual net helpfulness scores and tells us if the discrepancy is more than just the result of random chance. (You can read more about our statistical significance tests here). If the discrepancy is statistically significant it may indicate that the reviews are being unfairly targeted because people disagreed with the content of the review.