Update 10/3/2016: Amazon BANS Incentivized reviews!

We didn’t want to derail our Analysis of 7 million Amazon reviews: customers who receive free or discounted item much more likely to write positive review by going off on a tangent, so we decided to make this an entirely separate post.

The data shows that 86% of the time, the incentivized reviews give the same product a higher rating than the non-incentivized reviews.  But what about the other 14% of the time?  

When we use the term “non-incentivized reviews”, we’re referring to all reviews that lack the “free product for review” disclosure.  We aren’t magically limiting that selection to honest, verified full-price payers.  Just because a review is non-incentivized does not automatically make it trustworthy.

Here’s a deeper look into a product which has a higher rating from the non-incentivized than the incentivized reviews: #1 Brain Supplement Nootropic – Mind & Energy Booster – Clarity by Neovicta.  The first thing you’ll notice is that incentivized reviews make up 25% of the total reviews, and rate this product a 4.5 while the non-incentivized reviews rate it 4.7.  You may assume this is a counterexample to the data that shows incentivized reviews are typically higher, but on a closer look, it’s clear that the non-incentivized reviews are showing major problems of their own.

In our sample of 91 reviewers, 51 of them had reviewed other products by this same brand and rated this product a 4.9.  Only one of these reviewers was incentivized, meaning that over 65% of the non-incentivized reviewers are brand repeaters, rating the product much higher than the non-repeaters.

Furthermore, in this same sample of 91 reviewers, 32 of them have never written a verified purchase review.  Only one of these reviewers was incentivized, meaning that over 40% of non-incentivized reviewers have never written a verified purchase review, and have rated the product an average of 5.0 stars.

Lastly, looking at the top 10 Least Trusted Reviews, we see that none are incentivized:

If you entirely remove all the highly suspicious Brand Repeaters and the Never Verified Reviewers, the comparison looks like this:

Non-Incentivized Reviews: 3.4 stars

Incentivized Reviews: 4.7 stars

Obviously, this product has unnatural reviews beyond just the incentivized reviews.  It’s important to keep in mind that these highly-inflated, unnatural, yet non-incentivized reviews are still included in our aggregate comparison of incentivized vs. non-incentivized.  So when we say that after 7 million reviews, the non-incentivized reviews give an average of 4.36 stars, the actual rating from honest, non-incentivized reviewers could be much lower than that.

Leave a Reply