Browser Extensions 

vine-header-1

Amazon Vine: An Analysis of 30 Million Reviews Shows Vine Better than Incentivized

November 21st, 2016

 

On September 19th, 2016, we published a video showing that Incentivized Reviews (reviews including a disclaimer “I received this product for free or at a discount in exchange for my honest, unbiased review”) were highly biased, which quickly went viral, racking up over half a million views.

Two weeks later, on October 3rd, 2016, Amazon made an announcement that they were banning Incentivized Reviews.  In their announcement, they mentioned that the only acceptable way for vendors to give out free products for review was their own, in-house, Vine program.

Many agreed with Amazon’s decision and were happy to see them taking action by closing this loophole that was being exploited at record pace.  However, others were outraged, accusing Amazon of trying to make more money by forcing vendors to use their Vine program.  As usual, we wanted to examine the data to get a better understanding of what was really going on.

What the Data Tells Us

Since our last video, our dataset has grown to 30 million reviews, which is what we’ll be using to see how Vine reviews stack up to their Incentivized and non-incentivized (regular) counterparts.  The first thing we looked at was simply the average star rating for each type of review:

Screen Shot 2016-11-18 at 11.08.50 AM

So as you can see, Vine reviews appear to be the most critical of all.  However, things get a little interesting when you look at the distribution of star ratings:

Screen Shot 2016-11-20 at 12.36.23 PM

While Vine reviews still seem to be the most critical, you’ll notice a few things:

  • Vine reviews give a lot more 2, 3, and 4 star reviews than any other group.
  • Vine reviews are a lot more stingy with their 5-star reviews, being the only group to give out 5-stars less than half the time.
  • Vine reviews, just like incentivized reviews, are much less likely to be 1-star.
  • Over 20% of Vine reviews are critical (1, 2 or 3 stars)

Things Change When Controlling for Product

We had to narrow our dataset down considerably to control for product.  We only looked at products with at least 10 regular reviews and 10 Vine reviews, leaving us with 921 products and 2.5 million reviews, however only about 37,000 of them were vine reviews.

On average, the regular reviews rated the products 4.24 stars, while the Vine reviews rated the same products 4.39 stars, which is a .15 star increase!

Screen Shot 2016-11-18 at 11.14.50 AM

Out of the 951 products, 26% of them had a lower rating from their vine reviews, 12% of them had the same rating from their vine reviews, and 61% had a higher rating from their vine reviews:

Screen Shot 2016-11-20 at 12.37.34 PM

This doesn’t reflect too well on Vine reviews, however we wanted to stack them up against their incentivized counterparts while still controlling for product.

The next thing we did was narrow our dataset down even further.  We looked at only products that had at least 10 vine reviews, 10 incentivized reviews and 10 regular reviews.  We only found 107 products in our dataset that met this criteria.  We found that incentivized reviews are still the least critical group:

Screen Shot 2016-11-18 at 11.23.59 AM

So either way you look at it, vine reviews don’t appear to be nearly as biased as their incentivized counterparts, but still might not be perfectly representative of regular reviews.

Why are Vine Reviews Not Perfect?

Both Amazon Vine and the 3rd party Review Clubs operated with the same premise: you get free stuff to write a review about.  

So there might still be a few inherent biases at play.  

The first is that regular reviewers are more likely to choose to write a review after a negative experience with the product, while vine voices are required to review everything they get.  This could explain why we see regular reviewers give out 1-stars at three times the frequency of vine voices.

The next is still the psychological bias of viewing a product more favorably because you received it at no cost to yourself.  However, vine voices are required to pay taxes, and we’ll get to that in a minute.

Why Amazon Vine is Better Than 3rd Party Review Clubs

The real difference between Amazon Vine and the 3rd party review clubs is the manner in which they operate, and there’s 5 things that Amazon Vine does differently which help eliminate a lot of the bias.

  1. Accepting reviewers to their program.  Amazon Vine is invite-only, selecting only those who already have an established history of writing quality reviews on their platform.  On the other hand, 3rd party Review Clubs were eager to hand out memberships to anyone capable of submitting reviews to Amazon.
  2. Amazon does not allow the vendors to pick the reviewers.  This is absolutely crucial in preventing reviewers from posting exclusively 5-star reviews in order to keep getting more free stuff.
  3. Limiting the number of products a vendor can send out.  As you’re already aware, the 3rd party review clubs were completely out of control.  At one point, this product had over 8,000 incentivized reviews, but after the ban, we’re seeing that most were deleted.  The product with the most vine reviews was this one, and only had 423 vine reviews.
  4. Taxes.  Amazon Vine sends out 1099’s and requires everyone to report the fair market value of products received on their tax returns.  That may seem trivial, but consider an individual living in California making $55k a year – their marginal tax rate could be as high as 49.6%.  Meaning that if you receive a $1000 TV through Vine, you’ll have to pay $496 of taxes out of your own pocket.  On the other hand, we could not find any examples of 3rd party review clubs sending out 1099’s.  This likely resulted in 10’s or 100’s of millions of dollars of taxable income going unreported.
  5. Amazon does not allow for vendors to have any contact with vine voices, while many 3rd party review clubs connected the vendors with reviewers long before the product was even sent out.  This direct relationship is a massive conflict of interest if your goal is to create unbiased reviews.

So when Amazon made their announcement, many folks were criticizing them for forcing vendors through their own Vine program just so they could make more money, but it’s very unlikely that was the only reason.  While the data shows that Amazon Vine isn’t 100% perfect,  it’s objectively better than the 3rd party review clubs which were getting completely out of hand.

If vendors want to send out free products for review, they still have an option, however it’s not one that will give them 8,000 reviews in only a few months.

Possible Undesired Side-Effects

Now, one of the undesired side-effects of Amazon’s decision is that many vendors will simply choose to ignore Amazon’s TOS and continue getting incentivized reviews, but this time more discreetly and without the disclaimer.  

ReviewMeta is still an extremely powerful tool to help you identify unnatural patterns in the reviews for any product on Amazon.  Even without the disclaimer, incentivized reviews are easy for us to pick up on – you’ll just see the product failing other tests, for example: unverified purchases, rating trend, brand repeats, reviewer ease, phrase repetition, overlapping review history and more.

Simply copy and paste the amazon product URL into ReviewMeta.com or you can use our browser extension and we’ll give you a detailed report.


12 responses to “Amazon Vine: An Analysis of 30 Million Reviews Shows Vine Better than Incentivized”

  1. Sanpete says:

    Good to see this data crunched. Thanks!

    This may have been pointed out already somewhere, but Vine reviewers are more likely to know better than to leave a 1-star review because an item was delivered to the wrong house or the like. They get all deliveries directly from Amazon, while others may receive items from other sellers, which introduces more potential reasons for bad reviews, including fake or wrong products. It’s possible that such things explain the differences observed here.

  2. Bobby Cratchet says:

    Amazon Vine users are not required to post a review. I know a Viner and he does have the option to not post a review. That item will remain in his Items Awaiting Review list but the “free stuff” keeps on coming but he does keep a fairly steady stream of reviews trickling in. While a Vine Voice user that doesn’t leave reviews will probably exit sooner than later because Amazon is doing them a HUGE solid so reciprocation is expected but not required and the vendors are made aware of this fact prior to offering Vine users the opportunity to test their product.

    The tax burden is one that can possibly be avoided. They can keep their Estimated Tax Value below the $600 mark. They can order items with a smaller tax value. There are also some items that don’t have an estimated tax value. This might include food but he won’t tell me. He has said he’s seen some $0.00 Tax Values applied to unexpected items as well.

    The unfortunate part is that he has yet to complete a year of reviewing through the program so he doesn’t understand the tax percentage he will be responsible for but he has been researching it before he gets in too deep. This article suggests a possible 50% tax write up but as with anything tax related the final number is related to the Vine Voice member’s specific situation and his is quite unique. He is treating this as a hobby because he knows there is a 0% chance he will turn a profit nor is that his intent. His goal is simply improving his quality of life while hopefully doing the same for others through his reviews.

    • Sanpete says:

      Yes, food is (generally) assigned $0 ETV, also personal care items like soap, toothbrushes, etc. Amazon has never explained this, to my knowledge, but it may be because the retail or resale value of such items drops to essentially zero once they’re opened.

      Currently Amazon has no published rule about how many reviews must be completed to remain in good standing with Vine. In the past they varied from 75% or so to 100%. I think the current lack of explicit guidance has to do with not making Vine reviewers seem like employees to the IRS. Reviewers may have items they’ll never review, that have been on the list for years, with no repercussions I’m aware of.

      That might reduce the number of negative reviews.

      Many in Vine do treat it as a hobby for tax purposes, which means it’s not counted as income, but if you rack up high ETV totals (you can easily go over $10,000 if you’re busy) year after year the IRS might take a different view, judging from the IRS rules. Most Viners I’m aware of do consider ETVs in their selections.

  3. Mark Chopping says:

    I’ve done some work with the Amazon Review open dataset using an AWS Marketplace product from Kognitio, which looks at how Vine reviews compare to others, and also looks at the top reviewers and their participation in the Vine programme. See https://www.linkedin.com/pulse/100-shades-grey-other-amazon-review-discoveries-mark-chopping/ for more details – as this uses an open dataset stored in Amazon S3, and an AWS Marketplace product which is available for a few dollars per hour, you can easily do further investigation yourself.

    • Great write-up! The problem with that open dataset is that it only goes through 2015, so it’s hard to draw any recent trends from it.

      • Mark Chopping says:

        Yes, it’s a shame they haven’t updated it with more recent data – for example, would be good to look at the impact of removing incentivized reviews, which I believe is something Amazon implemented around March 2016.

  4. Robert Schmidt says:

    As a VIne reviewer, I find myself being more critical of Vine products than with other, non-Vine products. Hey, I feel that it is my responsibility, since I am a user of reviews for products that I purchase (or thought I was, until I read the reviews!). For non-Vine reviews, I can be as flippant, brief, or politicized as I want. I feel a different responsibility with my Vine reviews.

  5. LarryA2010 says:

    You say

    So as you can see, Vine reviews appear to be the most critical of all.

    Yet the graph clearly shows about 1/3 as many 1-star reviews as regular ones, an approximately equal number of 2-star reviews, and almost twice as many 3-star reviews.
    https://uploads.disquscdn.com/images/133dc35eb089f212e37c2857a04bfa903796872017313bb5a96a08a826590cd5.png

    What is your definition of most critical? Are 3-star reviews the cutoff? If they are then 1-, 2-, and 3-star reviews from Vine reviewers rack up 20.2% of the most critical reviews, and from Regular reviewers 17.6%. On the other hand, reviews from Incentivized reviewers total 4.7%.

  6. John says:

    What’s also missing from this analysis is the fact that Vine items tend to be higher end- brand name, advance copies of books, new releases, etc- and that members only select items they really want so by dint they’d be both higher quality and higher rated than some bootleg essential oil or plastic spatula set picked just to pad your stats.

  7. Greg Foley says:

    The major source of bias in Vine is probably how Amazon chooses to invite people. Based on your data it seems like they may invite people who rate higher than average reviewers.

Leave a Reply