Browser Extensions 

feat_res

Analysis of 7 million Amazon reviews: customers who receive free or discounted item much more likely to write positive review

June 29th, 2016

Update, 10/3/2016: Amazon BANS incentivized reviews two weeks after our video goes viral:

 

If you’ve read reviews on Amazon within the last few years, you’ve surely noticed a disclaimer at the bottom of many that look like this:

“I received this product for free or at a discount in exchange for my honest, unbiased review”

 

At ReviewMeta, we call these “Incentivized Reviews”.

Consumers have growing distrust and even disdain for incentivized reviews, especially when it seems every single one is a glowing 5-star review.  We wanted to confirm or deny this seemingly anecdotal opinion, so we analyzed 7 million reviews.  

After looking at over 7 million reviews, here’s what we see

We found that reviews containing language that would indicate the reviewer received the item for free or at a discount in exchange for a review (incentivized reviews) on average rate the product .38 stars higher than reviews that did not contain this disclosure (non-incentivized reviews).  

Non-Incentivized:

5,071,232 reviews (70%), 4.36 average rating

stars-nic-1

 

Incentivized:

2,174,802 reviews (30%), 4.74 average rating

stars-ic-1

 

With a 0.38 star difference, the impact may appear marginal, however the actual boost in average product rating can be substantial:

Considering that the average product on Amazon is rated around 4.4 stars, a boost from 4.36 to 4.74 stars can mean the difference between a mediocre product and a top rated product.  The graph below shows the percentile of products for each average rating:

Screen Shot 2016-06-27 at 9.55.25 PM

 

The distribution of ratings may be even more disturbing:

Incentivized reviewers are 12 times less likely to give a 1-star rating than non-incentivized reviews, and almost 4 times less likely to leave a critical review in general.

n-ic-key Non-Incentivized

ic-key Incentivized

Screen Shot 2016-06-27 at 4.49.27 PM

 

Controlling for product still shows a bias

An obvious criticism to the above data is that these reviews are for different products.  However, even when controlling for the product we see a similar trend.

We took a look at products that have at least 10 incentivized reviews AND 10 non-incentivized reviews (251 products with a total of 609,766 reviews).  On average, the incentivized reviewers gave the exact same product 0.29 stars higher than the non-incentivized reviewers:

stars-ic copy

 

86% of products received a higher average rating from their incentivized reviews than non-incentivized reviews:

Screen Shot 2016-06-29 at 8.54.04 PM

 

Here we can see the distribution of discrepancies between the incentivized average rating and the non-incentivized average rating:

n-ic-key Non-Incentivized Higher

ic-key Incentivized Higher

Screen Shot 2016-06-28 at 9.30.20 AM

 

Here are some products that exemplify incentivized reviewers giving a much higher rating than the non-incentivized reviewers:

We also took a closer look into a counterexample (a product that received a lower rating from its incentivized reviews than its non-incentivized reviews) and noticed that when this happens, it can be caused by the non-incentivized reviews being even more biased than the incentivized reviews.

When Incentivized Reviews are Lower than Non-Incentivized: A Possible Warning Sign

Those who write incentivized reviews tend to be less critical

We also examined the habits of reviewers who participate in these “free product for review” programs.  We only looked at reviewers who have written at least 10 reviews total.  We found that on average, reviewers who have written at least one incentivized review give an average of 4.56 stars, while reviewers who have never written any incentivized reviews give an average of 4.27 stars.

The chart below shows the percentage of reviewers for each average rating.  For example, 14% of reviewers with at least one incentivized review give an average of 4.9 stars for all products they review, while only 6% of reviewers with no incentivized reviews give an average of 4.9 stars for all products they review.

n-ic-key Reviewers With No Incentivized Reviews

ic-key Reviewers With At Least One Incentivized Review

Screen Shot 2016-06-27 at 6.02.13 PM

 

Furthermore, reviewers with at least one incentivized review have written an average of 232 reviews each while reviewers who have never written an incentivized review have written an average of 31 reviews each.  There appears to be a trend of high-volume reviewers who rate almost everything 5 stars.  Here are a few example profiles and current stats as of the time of writing:

  • myreview2: 845 reviews, every single one is 5-star.
  • Notavailableonhere: 758 reviews, every singe one is 5-star.
  • Amazon lover: This reviewer has 4,379 reviews.  The first review was written on March 23rd, 2015, so just shy of 10 reviews per day, every day for more than a year.  All are 5-star reviews with the exception of 9 that are 4-star, and 2 that are 3-star.  (The 3-star reviews did not even contain an incentivization disclosure!)

Volume of incentivized reviews is increasingly sharply

While we’ve only examined a sample of 7 million reviews, we’re still seeing that incentivized reviews are increasing in popularity at an alarming rate.  This graph shows the review volume during the last 48 months between May, 2012 and April, 2016:

n-ic-key Non-Incentivized

ic-key Incentivized

Screen Shot 2016-06-28 at 11.39.54 AM

 

2 years ago, incentivized reviews accounted for less than 2% of new reviews.  Since February of this year, they make up the majority of all new reviews on Amazon.

What about Amazon Vine?

Update 7/1/2016 – After several questions, we wanted to make a clarification:  Amazon Vine is Amazon’s in-house free-product for review system.  It is invite-only, often focuses on larger-value items and can even require the reviewers to return the item.  Amazon Vine reviews are NOT included in our incentivized group because they are NOT required to leave the “free product” disclaimer since their reviews are already labeled with the “Vine” badge.

Amazon Vine reviews make up roughly 4% of the reviews in our dataset, and give an average of 4.2 stars (compared to 4.74 from incentivized reviewers and 4.36 from non-incentivized), so at first glance, it seems to be much better controlled than their “review club” counterparts.  We’ll be doing an in-depth analysis on Vine reviews shortly, so stay tuned by subscribing to our mailing list, following us on Twitter and liking us on Facebook.

Where do these reviews come from, and why are they biased?

Incentivized reviews are a new and rapidly growing breed of review.  While some people label them as “fake”, incentivized reviews don’t technically deserve it because they are likely from real people who had a real experience with the actual product.  However, the data still shows they still have a bias – despite their claims to the contrary. Related: Unnatural Reviews and Why We Don’t Use the Word “Fake”

Dozens of “review clubs” (3rd party services) have popped up in the last few years to facilitate the exchange of free or discounted products for “honest” reviews.  This is fair game according to Amazon’s Terms of Service as long as the reviewer discloses this fact.

No matter what explanation you choose to believe, it’s hard to argue with the data.  Some might try to debunk our entire analysis by citing evidence that they (or a friend/family member) accept free products in exchange for reviews, and they always leave 100% honest feedback.  We aren’t calling them a liar; of course there will be “honest” reviewers who participate in these programs.  But this is the very definition of anecdotal evidence, and our analysis is on the bigger picture.

Now that we’ve established a clear bias in the data, we’re ready to dive into Part II: The Fundamental Problem with Review Clubs


  • Lord Cavendish

    Its hard to find a negative product review from Vine participants, ever.

    I ignore them.

    • That’s purely anecdotal. I would doubt that you’ve looked at millions of vine reviews like we have. 20% of them are either 1, 2 or 3 stars.

  • Malignant Juan

    I think Amazon picks only the positive reviews to be posted. I have spent thousands of dollars on Amazon. Buying pet food to tooth brushes. I was happy with the fast shipping and the return policy. Then I placed several reviews about dog food from Fromm company I was buying on Amazon. I stated as a review, each purchase, that I wish there was an expiration date on this food. It was a neutral review. Days later, Amazon sent me an email that they would not post this. Their suppression of my review request that sellers of dog food items be made to adhere to expiration dates is a BAD SIGN! IT CONCERNS ME THAT AMAZON IS PERMITTING SELLERS OF DOG FOOD WHO MIGHT BE SELLING EXPIRED FOOD.

  • Felicia

    Sadly I’m now coming across far too many products of which the majority of existing reviews are from incentivised (UK customer) reviews. Some products don’t have a single non-incentivised review. Like someone mentioned below, I just skip those products altogether. My opinion is the review system has become severely compromised. A possible solution: Amazon needs to revise their system in a way that separates and highlights reviews by those who have received a product at a discount or for free in exchange for a review, and perhaps colour code the star values. For example, yellow stars = non-incentivised, red = incentivised, orange = a combination of the two. Thereby leaving the consumer to choose for themselves.

  • ACEamI

    Meh. People who received product in exchange for a review are going to leave a review.

    In general, people who pay full price are more likely to leave a review if they are unhappy with the product. Those who love it don’t often leave one.

    So…. the “bias” may only be the requirement to do so.

    • This is one of many theories, however Amazon has already prohibited these reviews.

  • Well done guys, nice reporting and a lot of interesting data about reviews. Seems like Amazon reacted with a change of their guidelines concerning how to gain reviews. We’ve wrote an german blog article about this topic and also mentioned your survey. Maybe interesting for some german user here: http://www.intomarkets.com/neue-amazon-richtlinie-keine-rezensionen-mehr-durch-produkttester-erlaubt/

  • Peter

    i literally pass over products once i see a review with the disclaimer.

  • Lucas du Toit

    Does that bias change depending on the price of the item? I’d like love to see that data!

    • Bhima

      I think its clear that it does and we don’t really need data to prove it. You buy a product for 1/2 what you think its worth, find a few quirks but accept them as a few minor blemishes to a good product at a great price. If you paid alot of money, especially compared to competiting offerings and there are flaws, your expectations change commensurate with how you feel the product price is wrong for the item.

  • tomclarke

    Thank you for researching this. I’ve been getting really annoyed with these reviews. On some products, all or nearly all the reviews are incentivized and not everyone realizes.