ReviewMeta Analysis Test: Incentivized Reviews
Update 10/3/2016: Amazon BANS incentivized reviews!
If you’ve looked through a lot of Amazon reviews you may have seen recurring phrases like “I have received this product for a discount in exchange for my honest review.” Sometimes these reviews will start by telling you this, sometimes they’ll end with it and sometimes they’ll sneak this sentence right in the middle of an unrelated paragraph. You might have also noticed that these reviews tend to be clustered; you’ll see a number of reviews like this for one product, but then see none of these reviews for other products. So what is happening here?
According to Amazon’s Review Guidelines it is not permitted to post reviews in exchange for compensation. This means that a company cannot pay someone to write a review of their product. However, Amazon makes an exception to this rule. Companies are allowed to give out free or discounted items to customers in exchange for a review of that product, on the condition that the reviewer discloses this within their review.
There is a whole industry built around this loophole. Dozens of third-party companies have been created to act as middlemen in this exchange, bringing together brands who need reviews and reviewers who want free stuff. However, these third party companies have no vested interest in trying to create high quality reviews. They sell themselves to brands by claiming that they can dramatically help sales by attracting more reviewers to write reviews. Furthermore, they aren’t very selective about which reviewers they accept to their programs: the requirements usually amount to having an Amazon account and disclosing that you received the product for free within the body of the review.
The major problem with these third party services is that they can pick which reviewers receive free items based on their previous reviews. Write one too many negative reviews and you’ll risk never getting picked for a review again.
When a reviewer receives a free or discounted product in exchange for a review, they are writing an incentivized review. Here is why we think incentivized reviews are problematic:
- Motive: If you are writing a review for a product that you paid for, chances are that you are writing the review because you want to help other customers make an informed decision about the product. However, when reviewers write reviews in exchange for free goods, they are writing them primarily to receive the free product – it’s just a step they are required to take in order to receive the product they want. This has lead to some reviewers becoming “super reviewers”, who will review multiple products a day, some reviewing thousands of products a year.
- Quid-pro-quo: Reviewing a product you received for free creates a quid-pro-quo situation. Reviewers might feel obligated to write a positive review since they feel indebted to the company who sent them the free product. You scratch my back (give me a free product) and I’ll scratch yours (give you a good review). Often times this will be implicit, “they might not send me more products if I write a negative a review” and more rarely this demand will be explicit “you have to write a positive review if you want this product”.
- Reviewer Bias: Even if a reviewer is not explicitly told to write a positive review or feel an obligation to write a positive review, receiving a product for free creates a subconscious positive bias. Having a company give you a product for free will make you think higher of the company. And since you are getting the product for free, or heavily discounted, you don’t have the same amount of financial skin in the game as reviewers who bought the product for the retail price. Because of this lack of financial investment incentivized reviewers are less likely critical than reviewers who actually bought the product.
- Selection Bias: Often times with third party companies, if you write negative reviews, you won’t get picked to receive more free products; they will kick you off the mailing list. This creates a perfect breeding ground for positive reviews. Third party companies control the lists and are constantly refining it to make sure that it only consists of positive reviewers.
All of these factors mean that incentivized reviewers are not necessarily representative of a typical reviewer and can unnaturally influence a product’s average rating. The overall data seems to agree with this fact. Out of all the Amazon reviews we’ve looked at, incentivized reviews give an average of 4.8 stars while non-incentivized reviews give an average of 4.4 stars.
Luckily for us these incentivized reviews are required to disclose that they have received the product in exchange for a review. So at ReviewMeta we developed a function that can detect language that would indicate this disclosure and label these reviews as “incentivized”. We then tally them up to see what percentage of the overall reviews they make up. While it isn’t uncommon to see a small percentage of these reviews, an excessively high number can trigger a warning or failure of this test. We then compare the average rating of the incentivized reviews to all other reviews. If we see that incentivized reviews have a higher average rating than all other reviews, we’ll check to see if this discrepancy is statistically significant. This means that we run the data through an equation that takes into account the total number of reviews along with the variance of the individual ratings and tells us if the discrepancy is more than just the result of random chance. (You can read more about our statistical significance tests here). If we do find a statistically significant difference, it is a strong indicator that the incentivized reviews are not benign, but have been affected by biases and are unnaturally influencing the product’s overall rating.