ReviewMeta Analysis Test: Incentivized Reviews
Update 10/3/2016: Amazon BANS incentivized reviews!
If you’ve looked through a lot of Amazon reviews you may have seen recurring phrases like “I have received this product for a discount in exchange for my honest review.” Sometimes these reviews will start by telling you this, sometimes they’ll end with it and sometimes they’ll sneak this sentence right in the middle of an unrelated paragraph. You might have also noticed that these reviews tend to be clustered; you’ll see a number of reviews like this for one product, but then see none of these reviews for other products. So what is happening here?
According to Amazon’s Review Guidelines it is not permitted to post reviews in exchange for compensation. This means that a company cannot pay someone to write a review of their product. However, Amazon makes an exception to this rule. Companies are allowed to give out free or discounted items to customers in exchange for a review of that product, on the condition that the reviewer discloses this within their review.
There is a whole industry built around this loophole. Dozens of third-party companies have been created to act as middlemen in this exchange, bringing together brands who need reviews and reviewers who want free stuff. However, these third party companies have no vested interest in trying to create high quality reviews. They sell themselves to brands by claiming that they can dramatically help sales by attracting more reviewers to write reviews. Furthermore, they aren’t very selective about which reviewers they accept to their programs: the requirements usually amount to having an Amazon account and disclosing that you received the product for free within the body of the review.
The major problem with these third party services is that they can pick which reviewers receive free items based on their previous reviews. Write one too many negative reviews and you’ll risk never getting picked for a review again.
When a reviewer receives a free or discounted product in exchange for a review, they are writing an incentivized review. Here is why we think incentivized reviews are problematic:
- Motive: If you are writing a review for a product that you paid for, chances are that you are writing the review because you want to help other customers make an informed decision about the product. However, when reviewers write reviews in exchange for free goods, they are writing them primarily to receive the free product – it’s just a step they are required to take in order to receive the product they want. This has lead to some reviewers becoming “super reviewers”, who will review multiple products a day, some reviewing thousands of products a year.
- Quid-pro-quo: Reviewing a product you received for free creates a quid-pro-quo situation. Reviewers might feel obligated to write a positive review since they feel indebted to the company who sent them the free product. You scratch my back (give me a free product) and I’ll scratch yours (give you a good review). Often times this will be implicit, “they might not send me more products if I write a negative a review” and more rarely this demand will be explicit “you have to write a positive review if you want this product”.
- Reviewer Bias: Even if a reviewer is not explicitly told to write a positive review or feel an obligation to write a positive review, receiving a product for free creates a subconscious positive bias. Having a company give you a product for free will make you think higher of the company. And since you are getting the product for free, or heavily discounted, you don’t have the same amount of financial skin in the game as reviewers who bought the product for the retail price. Because of this lack of financial investment incentivized reviewers are less likely critical than reviewers who actually bought the product.
- Selection Bias: Often times with third party companies, if you write negative reviews, you won’t get picked to receive more free products; they will kick you off the mailing list. This creates a perfect breeding ground for positive reviews. Third party companies control the lists and are constantly refining it to make sure that it only consists of positive reviewers.
All of these factors mean that incentivized reviewers are not necessarily representative of a typical reviewer and can unnaturally influence a product’s average rating. The overall data seems to agree with this fact. Out of all the Amazon reviews we’ve looked at, incentivized reviews give an average of 4.8 stars while non-incentivized reviews give an average of 4.4 stars.
Luckily for us these incentivized reviews are required to disclose that they have received the product in exchange for a review. So at ReviewMeta we developed a function that can detect language that would indicate this disclosure and label these reviews as “incentivized”. We then tally them up to see what percentage of the overall reviews they make up. While it isn’t uncommon to see a small percentage of these reviews, an excessively high number can trigger a warning or failure of this test. We then compare the average rating of the incentivized reviews to all other reviews. If we see that incentivized reviews have a higher average rating than all other reviews, we’ll check to see if this discrepancy is statistically significant. This means that we run the data through an equation that takes into account the total number of reviews along with the variance of the individual ratings and tells us if the discrepancy is more than just the result of random chance. (You can read more about our statistical significance tests here). If we do find a statistically significant difference, it is a strong indicator that the incentivized reviews are not benign, but have been affected by biases and are unnaturally influencing the product’s overall rating.
I see a HUGE issue with your statistical test. Repeated checks do not take into account for books that Amazon allows free or discounted copies to reviewers for editorial review. Your blanket test that anyone who is gifted a product is skewed. You need to carefully review Amazon Community Standards into effect and not punish legitimate editorial reviews for published works that ARE ALLOWED!
Look under Promotions and Commercial Solicitations. https://www.amazon.com/gp/help/customer/display.html?nodeId=201929730
Hello My_Own_Bookshelves –
Thank you for your feedback. There’s a few things to take into consideration here:
1. The ReviewMeta.com algorithm is not based on the Amazon TOS. Regardless of the TOS, ReviewMeta.com believes that in some situations, incentivized reviews on books can still be biased.
When ReviewMeta.com launched, all incentivized reviews were permitted under the Amazon’s TOS. Even though they were not against the TOS, I still created a test to identify them and devalue them if they appeared to be significantly affecting the overall rating or review count for that particular product. I believed that regardless of the TOS, if they appeared to be significantly impacting the overall rating or review count for a particular product, the incentivized reviews for that product should be devalued. When Amazon changed their policy, nothing in my algorithm changed.
2. As with all tests, the amount that we devalue reviews with any trait is determined on a product-by-product basis. For some products, we don’t devalue their incentivized reviews at all – as long as they aren’t significantly impacting the overall rating or review count for that product. If we devalue incentivized reviews for a specific product, it means that we detected there was significant impact on the product rating or review count from the incentivized reviews.
3. As with all tests, if you do not feel like the incentivized reviews should be devalued, you can adjust the weight of them and recalculate the adjusted product rating and review count to your liking.
I am in no way pretending that the ReviewMeta.com algorithm is perfect in every way, and that’s why there is a massive disclaimer at the top of every page for all first-time visitors until you click the “I understand” button.
In order to make reviewmeta compatible with German Amazon reviews, I recommend you look for these phrases – they are virtually always present in incentivized reviews:
“vergünstigt” – should have a very low false positive rate. Means they received the product with a discount.
“Produkttest” “zu Testzwecken” – those are the “tests” that people claim to have received the product for
“unvoreingenommene” – unbiased. Yeah right.
“ehrlich” and “Bewertung” – alone, this might not be sufficient, but combined with any other words above, this should raise a red flag.
“kostenlos” – means “free”. Same as above – alone this might not be enough, but combine it with any other words above, and you got a textbook incentivized review.
Pretty much all bought reviews (that admit to being bought, at least) have these phrases in them, so they should pose a low-risk way to make reviewmeta more valuable for your German speaking audience (as the other ranking methods you provide already work).
Thanks! We’ve got all of those in the system except for Testzwecken – which I just added now.
http://a.co/3ZEIuwr
has an Incentivized Review and your system didn’t detect it.
it says “This product was received at no cost for review and inspection purposes,”
Our algorithm for detecting incentivized text is about 99.5% accurate. So it’s not impossible for our system to miss a few in our dataset of over 100,000,000 reviews.
Could you please add a separate test for Vine reviews? Look at Amazon product B015ZDSM9M, for example – lots of 5 star and 4 star reviews from Vine reviewers who never tried the product. Perhaps introduce a separate test for “Reviewer did not try product.”
This is a negative side effect of Vine reviewers having a deadline to submit a review, and products like snow blowers, fire extinguishers, first aid equipment etc. can’t really be reviewed until the need for them arises. The ethical thing to do would be to return the product, but lots of people just write a 5 star review so they can keep the product.