Browser Extensions 

vine-header-1

Amazon Vine: An Analysis of 30 Million Reviews Shows Vine Better than Incentivized

November 21st, 2016

 

On September 19th, 2016, we published a video showing that Incentivized Reviews (reviews including a disclaimer “I received this product for free or at a discount in exchange for my honest, unbiased review”) were highly biased, which quickly went viral, racking up over half a million views.

Two weeks later, on October 3rd, 2016, Amazon made an announcement that they were banning Incentivized Reviews.  In their announcement, they mentioned that the only acceptable way for vendors to give out free products for review was their own, in-house, Vine program.

Many agreed with Amazon’s decision and were happy to see them taking action by closing this loophole that was being exploited at record pace.  However, others were outraged, accusing Amazon of trying to make more money by forcing vendors to use their Vine program.  As usual, we wanted to examine the data to get a better understanding of what was really going on.

What the Data Tells Us

Since our last video, our dataset has grown to 30 million reviews, which is what we’ll be using to see how Vine reviews stack up to their Incentivized and non-incentivized (regular) counterparts.  The first thing we looked at was simply the average star rating for each type of review:

Screen Shot 2016-11-18 at 11.08.50 AM

So as you can see, Vine reviews appear to be the most critical of all.  However, things get a little interesting when you look at the distribution of star ratings:

Screen Shot 2016-11-20 at 12.36.23 PM

While Vine reviews still seem to be the most critical, you’ll notice a few things:

  • Vine reviews give a lot more 2, 3, and 4 star reviews than any other group.
  • Vine reviews are a lot more stingy with their 5-star reviews, being the only group to give out 5-stars less than half the time.
  • Vine reviews, just like incentivized reviews, are much less likely to be 1-star.
  • Over 20% of Vine reviews are critical (1, 2 or 3 stars)

Things Change When Controlling for Product

We had to narrow our dataset down considerably to control for product.  We only looked at products with at least 10 regular reviews and 10 Vine reviews, leaving us with 921 products and 2.5 million reviews, however only about 37,000 of them were vine reviews.

On average, the regular reviews rated the products 4.24 stars, while the Vine reviews rated the same products 4.39 stars, which is a .15 star increase!

Screen Shot 2016-11-18 at 11.14.50 AM

Out of the 951 products, 26% of them had a lower rating from their vine reviews, 12% of them had the same rating from their vine reviews, and 61% had a higher rating from their vine reviews:

Screen Shot 2016-11-20 at 12.37.34 PM

This doesn’t reflect too well on Vine reviews, however we wanted to stack them up against their incentivized counterparts while still controlling for product.

The next thing we did was narrow our dataset down even further.  We looked at only products that had at least 10 vine reviews, 10 incentivized reviews and 10 regular reviews.  We only found 107 products in our dataset that met this criteria.  We found that incentivized reviews are still the least critical group:

Screen Shot 2016-11-18 at 11.23.59 AM

So either way you look at it, vine reviews don’t appear to be nearly as biased as their incentivized counterparts, but still might not be perfectly representative of regular reviews.

Why are Vine Reviews Not Perfect?

Both Amazon Vine and the 3rd party Review Clubs operated with the same premise: you get free stuff to write a review about.  

So there might still be a few inherent biases at play.  

The first is that regular reviewers are more likely to choose to write a review after a negative experience with the product, while vine voices are required to review everything they get.  This could explain why we see regular reviewers give out 1-stars at three times the frequency of vine voices.

The next is still the psychological bias of viewing a product more favorably because you received it at no cost to yourself.  However, vine voices are required to pay taxes, and we’ll get to that in a minute.

Why Amazon Vine is Better Than 3rd Party Review Clubs

The real difference between Amazon Vine and the 3rd party review clubs is the manner in which they operate, and there’s 5 things that Amazon Vine does differently which help eliminate a lot of the bias.

  1. Accepting reviewers to their program.  Amazon Vine is invite-only, selecting only those who already have an established history of writing quality reviews on their platform.  On the other hand, 3rd party Review Clubs were eager to hand out memberships to anyone capable of submitting reviews to Amazon.
  2. Amazon does not allow the vendors to pick the reviewers.  This is absolutely crucial in preventing reviewers from posting exclusively 5-star reviews in order to keep getting more free stuff.
  3. Limiting the number of products a vendor can send out.  As you’re already aware, the 3rd party review clubs were completely out of control.  At one point, this product had over 8,000 incentivized reviews, but after the ban, we’re seeing that most were deleted.  The product with the most vine reviews was this one, and only had 423 vine reviews.
  4. Taxes.  Amazon Vine sends out 1099’s and requires everyone to report the fair market value of products received on their tax returns.  That may seem trivial, but consider an individual living in California making $55k a year – their marginal tax rate could be as high as 49.6%.  Meaning that if you receive a $1000 TV through Vine, you’ll have to pay $496 of taxes out of your own pocket.  On the other hand, we could not find any examples of 3rd party review clubs sending out 1099’s.  This likely resulted in 10’s or 100’s of millions of dollars of taxable income going unreported.
  5. Amazon does not allow for vendors to have any contact with vine voices, while many 3rd party review clubs connected the vendors with reviewers long before the product was even sent out.  This direct relationship is a massive conflict of interest if your goal is to create unbiased reviews.

So when Amazon made their announcement, many folks were criticizing them for forcing vendors through their own Vine program just so they could make more money, but it’s very unlikely that was the only reason.  While the data shows that Amazon Vine isn’t 100% perfect,  it’s objectively better than the 3rd party review clubs which were getting completely out of hand.

If vendors want to send out free products for review, they still have an option, however it’s not one that will give them 8,000 reviews in only a few months.

Possible Undesired Side-Effects

Now, one of the undesired side-effects of Amazon’s decision is that many vendors will simply choose to ignore Amazon’s TOS and continue getting incentivized reviews, but this time more discreetly and without the disclaimer.  

ReviewMeta is still an extremely powerful tool to help you identify unnatural patterns in the reviews for any product on Amazon.  Even without the disclaimer, incentivized reviews are easy for us to pick up on – you’ll just see the product failing other tests, for example: unverified purchases, rating trend, brand repeats, reviewer ease, phrase repetition, overlapping review history and more.

Simply copy and paste the amazon product URL into ReviewMeta.com or you can use our browser extension and we’ll give you a detailed report.


  • Robert Schmidt

    As a VIne reviewer, I find myself being more critical of Vine products than with other, non-Vine products. Hey, I feel that it is my responsibility, since I am a user of reviews for products that I purchase (or thought I was, until I read the reviews!). For non-Vine reviews, I can be as flippant, brief, or politicized as I want. I feel a different responsibility with my Vine reviews.

  • LarryA2010

    You say

    So as you can see, Vine reviews appear to be the most critical of all.

    Yet the graph clearly shows about 1/3 as many 1-star reviews as regular ones, an approximately equal number of 2-star reviews, and almost twice as many 3-star reviews.
    https://uploads.disquscdn.com/images/133dc35eb089f212e37c2857a04bfa903796872017313bb5a96a08a826590cd5.png

    What is your definition of most critical? Are 3-star reviews the cutoff? If they are then 1-, 2-, and 3-star reviews from Vine reviewers rack up 20.2% of the most critical reviews, and from Regular reviewers 17.6%. On the other hand, reviews from Incentivized reviewers total 4.7%.

    • Over 20% of Vine reviews are critical (1, 2 or 3 stars). This is more than both regular or incentivized reviews.

  • John

    What’s also missing from this analysis is the fact that Vine items tend to be higher end- brand name, advance copies of books, new releases, etc- and that members only select items they really want so by dint they’d be both higher quality and higher rated than some bootleg essential oil or plastic spatula set picked just to pad your stats.

    • This actually is addressed. When we “control for product”, we’re literally looking at vine reviews compared to the non-vine reviews on the exact same product.

  • The major source of bias in Vine is probably how Amazon chooses to invite people. Based on your data it seems like they may invite people who rate higher than average reviewers.