Amazon Vine: An Analysis of 30 Million Reviews Shows Vine Better than IncentivizedNovember 21st, 2016
On September 19th, 2016, we published a video showing that Incentivized Reviews (reviews including a disclaimer “I received this product for free or at a discount in exchange for my honest, unbiased review”) were highly biased, which quickly went viral, racking up over half a million views.
Two weeks later, on October 3rd, 2016, Amazon made an announcement that they were banning Incentivized Reviews. In their announcement, they mentioned that the only acceptable way for vendors to give out free products for review was their own, in-house, Vine program.
Many agreed with Amazon’s decision and were happy to see them taking action by closing this loophole that was being exploited at record pace. However, others were outraged, accusing Amazon of trying to make more money by forcing vendors to use their Vine program. As usual, we wanted to examine the data to get a better understanding of what was really going on.
What the Data Tells Us
Since our last video, our dataset has grown to 30 million reviews, which is what we’ll be using to see how Vine reviews stack up to their Incentivized and non-incentivized (regular) counterparts. The first thing we looked at was simply the average star rating for each type of review:
So as you can see, Vine reviews appear to be the most critical of all. However, things get a little interesting when you look at the distribution of star ratings:
While Vine reviews still seem to be the most critical, you’ll notice a few things:
- Vine reviews give a lot more 2, 3, and 4 star reviews than any other group.
- Vine reviews are a lot more stingy with their 5-star reviews, being the only group to give out 5-stars less than half the time.
- Vine reviews, just like incentivized reviews, are much less likely to be 1-star.
- Over 20% of Vine reviews are critical (1, 2 or 3 stars)
Things Change When Controlling for Product
We had to narrow our dataset down considerably to control for product. We only looked at products with at least 10 regular reviews and 10 Vine reviews, leaving us with 921 products and 2.5 million reviews, however only about 37,000 of them were vine reviews.
On average, the regular reviews rated the products 4.24 stars, while the Vine reviews rated the same products 4.39 stars, which is a .15 star increase!
Out of the 951 products, 26% of them had a lower rating from their vine reviews, 12% of them had the same rating from their vine reviews, and 61% had a higher rating from their vine reviews:
This doesn’t reflect too well on Vine reviews, however we wanted to stack them up against their incentivized counterparts while still controlling for product.
The next thing we did was narrow our dataset down even further. We looked at only products that had at least 10 vine reviews, 10 incentivized reviews and 10 regular reviews. We only found 107 products in our dataset that met this criteria. We found that incentivized reviews are still the least critical group:
So either way you look at it, vine reviews don’t appear to be nearly as biased as their incentivized counterparts, but still might not be perfectly representative of regular reviews.
Why are Vine Reviews Not Perfect?
Both Amazon Vine and the 3rd party Review Clubs operated with the same premise: you get free stuff to write a review about.
So there might still be a few inherent biases at play.
The first is that regular reviewers are more likely to choose to write a review after a negative experience with the product, while vine voices are required to review everything they get. This could explain why we see regular reviewers give out 1-stars at three times the frequency of vine voices.
The next is still the psychological bias of viewing a product more favorably because you received it at no cost to yourself. However, vine voices are required to pay taxes, and we’ll get to that in a minute.
Why Amazon Vine is Better Than 3rd Party Review Clubs
The real difference between Amazon Vine and the 3rd party review clubs is the manner in which they operate, and there’s 5 things that Amazon Vine does differently which help eliminate a lot of the bias.
- Accepting reviewers to their program. Amazon Vine is invite-only, selecting only those who already have an established history of writing quality reviews on their platform. On the other hand, 3rd party Review Clubs were eager to hand out memberships to anyone capable of submitting reviews to Amazon.
- Amazon does not allow the vendors to pick the reviewers. This is absolutely crucial in preventing reviewers from posting exclusively 5-star reviews in order to keep getting more free stuff.
- Limiting the number of products a vendor can send out. As you’re already aware, the 3rd party review clubs were completely out of control. At one point, this product had over 8,000 incentivized reviews, but after the ban, we’re seeing that most were deleted. The product with the most vine reviews was this one, and only had 423 vine reviews.
- Taxes. Amazon Vine sends out 1099’s and requires everyone to report the fair market value of products received on their tax returns. That may seem trivial, but consider an individual living in California making $55k a year – their marginal tax rate could be as high as 49.6%. Meaning that if you receive a $1000 TV through Vine, you’ll have to pay $496 of taxes out of your own pocket. On the other hand, we could not find any examples of 3rd party review clubs sending out 1099’s. This likely resulted in 10’s or 100’s of millions of dollars of taxable income going unreported.
- Amazon does not allow for vendors to have any contact with vine voices, while many 3rd party review clubs connected the vendors with reviewers long before the product was even sent out. This direct relationship is a massive conflict of interest if your goal is to create unbiased reviews.
So when Amazon made their announcement, many folks were criticizing them for forcing vendors through their own Vine program just so they could make more money, but it’s very unlikely that was the only reason. While the data shows that Amazon Vine isn’t 100% perfect, it’s objectively better than the 3rd party review clubs which were getting completely out of hand.
If vendors want to send out free products for review, they still have an option, however it’s not one that will give them 8,000 reviews in only a few months.
Possible Undesired Side-Effects
Now, one of the undesired side-effects of Amazon’s decision is that many vendors will simply choose to ignore Amazon’s TOS and continue getting incentivized reviews, but this time more discreetly and without the disclaimer.
ReviewMeta is still an extremely powerful tool to help you identify unnatural patterns in the reviews for any product on Amazon. Even without the disclaimer, incentivized reviews are easy for us to pick up on – you’ll just see the product failing other tests, for example: unverified purchases, rating trend, brand repeats, reviewer ease, phrase repetition, overlapping review history and more.