When Incentivized Reviews are Lower than Non-Incentivized: A Possible Warning Sign
Update 10/3/2016: Amazon BANS Incentivized reviews!
We didn’t want to derail our Analysis of 7 million Amazon reviews: customers who receive free or discounted item much more likely to write positive review by going off on a tangent, so we decided to make this an entirely separate post.
The data shows that 86% of the time, the incentivized reviews give the same product a higher rating than the non-incentivized reviews. But what about the other 14% of the time?
When we use the term “non-incentivized reviews”, we’re referring to all reviews that lack the “free product for review” disclosure. We aren’t magically limiting that selection to honest, verified full-price payers. Just because a review is non-incentivized does not automatically make it trustworthy.
Here’s a deeper look into a product which has a higher rating from the non-incentivized than the incentivized reviews: #1 Brain Supplement Nootropic – Mind & Energy Booster – Clarity by Neovicta. The first thing you’ll notice is that incentivized reviews make up 25% of the total reviews, and rate this product a 4.5 while the non-incentivized reviews rate it 4.7. You may assume this is a counterexample to the data that shows incentivized reviews are typically higher, but on a closer look, it’s clear that the non-incentivized reviews are showing major problems of their own.
In our sample of 91 reviewers, 51 of them had reviewed other products by this same brand and rated this product a 4.9. Only one of these reviewers was incentivized, meaning that over 65% of the non-incentivized reviewers are brand repeaters, rating the product much higher than the non-repeaters.
Furthermore, in this same sample of 91 reviewers, 32 of them have never written a verified purchase review. Only one of these reviewers was incentivized, meaning that over 40% of non-incentivized reviewers have never written a verified purchase review, and have rated the product an average of 5.0 stars.
Lastly, looking at the top 10 Least Trusted Reviews, we see that none are incentivized:
If you entirely remove all the highly suspicious Brand Repeaters and the Never Verified Reviewers, the comparison looks like this:
Non-Incentivized Reviews: 3.4 stars
Incentivized Reviews: 4.7 stars
Obviously, this product has unnatural reviews beyond just the incentivized reviews. It’s important to keep in mind that these highly-inflated, unnatural, yet non-incentivized reviews are still included in our aggregate comparison of incentivized vs. non-incentivized. So when we say that after 7 million reviews, the non-incentivized reviews give an average of 4.36 stars, the actual rating from honest, non-incentivized reviewers could be much lower than that.