Browser Extensions 

verified

ReviewMeta Analysis Test: Unverified Purchases

April 27th, 2016

The verified purchaser system is simple. If you purchase a product from an online vendor and then go on to review it, the vendor will add a flair to your review saying “verified purchaser”. Since most sites will allow users to post reviews whether or not they have purchased the product from the site, the verified purchaser flair adds another level of authenticity and legitimacy to those reviews. When you read a review with a verified purchaser flair you know that the reviewer received the product from the vendor.  This is one of many safeguards vendors employ in order to protect their customers by stopping some brands from artificially inflating their product’s rating and reputation.  

Regardless, you’ll still see some brands stuffing a disproportional amount of unverified reviews in order to boost their rating.  While some amount of unverified reviews are to be expected for any product, it can be a red flag if their percentage is too high, or if the unverified reviews are telling a story that is vastly different from the verified reviews.

Although we know that verified reviewers most likely received the product, the flair alone is not an indication of a trustworthy review. Some brands will select reviewers and give them a discounted or free product in exchange for an “honest review”. Despite being free or heavily discounted, some sites may still label these reviews as verified reviews.  Furthermore, some brands will offer a “trial size” or generate coupons or gift codes for 100% off, allowing some to achieve the “verified purchaser” flair on fraudulent reviews. We have even heard of brands going so far as to create fake distribution centers where they route all fake orders that never get shipped out.

At ReviewMeta we are able to learn more about a product’s reviews by examining the data that the verified purchaser system creates. While it’s perfectly normal to see some unverified reviews, an excessive percentage can trigger a warning or failure for this test . Furthermore, if the unverified reviews have a higher average rating than verified reviews, we’ll check to see if this discrepancy is statistically significant.  This means that we run the data through an equation that takes into account the total number of reviews along with the variance of the individual ratings and tells us if the discrepancy is more than just the result of random chance.  (You can read more about our statistical significance tests here).  If the unverified reviews have a significantly higher average rating than verified reviews, it’s a strong indicator that the unverified reviews are influencing the overall rating. This would be a strong indicator that these reviewers are not benign, but in fact, abusing the system.


8 responses to “ReviewMeta Analysis Test: Unverified Purchases”

  1. Taufiq says:

    Hi There Review Meta,

    Your website intrigued me and I ended up actually analyzing my own listing. One of the areas that your “algorithm” failed my listing was on the unverified purchases front. I just wanted to put forth my thought on why the score is incorrect and how you can improve the algorithm.

    1. I have a total of 2% unverfified purchase reviews (1 out of 49).

    2. The one review that I have from an unverified purchase has a score of 1 whilst the remaining give me a score of 4.7. Whilst I understand that there is a statistical difference, the key point to note is that the difference is not in my favour and thus it should not be a fail.

    3. Thus I believe you need to keep an additional check on the score on whether the difference is in favour of the listing or against it.

    • A single failed test does not suggest that you have created fake reviews, but points out that there is an abnormality in the data. All of what you stated is clearly visible in the report under that test. If the one review that “failed” that test is thrown out, it should actually help improve your adjusted rating.

  2. Nero Se says:

    Dear Review Meta,

    this is a long read but it is necessary to get my point across about the “unverified Badge”
    Recently I came across quite a few of my Reviews (I had 100% verified Purchases) was missing the “verified” badge I first noticed this infact on RM site .

    I contacted amazon thinking this was a mistake their emailed reply was ” An Amazon Verified Purchase means that the person writing the review purchased the product at Amazon and did not receive the product at a deep discount. There is no limit on Amazon Verified Purchase reviews”
    again thinking they hadn’t read my email properly I wrote in details with the order numbers
    as vast majority of the products sold by amazon UK quite a few were in fact amazon basics products.
    this time Amazon’s Reply was flabbergasting !!
    “With regards to the other orders, a discount has been applied to them or they’ve been partially paid for with a gift card, thus do not qualify for the “Verified Purchase” badge, amazingly they mentioned because I used gift cards -which were purchased through when ever they had a gift card bonus promotion amazon , some orders were unverifed because i used a gift card balance(purchased through them- there were others converted in to GC through cashback sites , but at least £100 in vouchers purchased through amazon.

    I replied saying the Discounts were prime students discounts for basics products, while others the discount was gained by the promotions box on the product page it self including the vouchers that are available to collect through amazon product page itself which are available for all users & Gift Cards were purchased mostly through Amazon.
    however they were not willing to help.

    My question is in a case as above where all of my review are in fact genuine how does Review meta analyse this ??
    there must be a lot of genuine reviewers like me stung out by this policy. where the fakers get nearly all verified purchase badges .i know this because I have listened to the recent BBC investigation (http://www.bbc.co.uk/news/technology-43907695) on which Review meta participated . I have searched these groups as revealed in it and found all of them give the fakers a money by pay pal there by must be achieving 100 % verified badges as no genuine gift cards or discounts are used .
    so this new method by amazon it seems will only affect genuine reviews not fakers .
    what impact will this have on the “trust rating” and in fact the overall product?

    • Hello Nero-

      The verified purchaser test is only one of the 15 tests that we run on the data, and whether or not a review has this badge does not impact the other tests. Furthermore, we do not devalue ALL reviews which do not have this badge. We only devalue the unverified reviews of a specific product IF they seem to have a substantial impact on the product rating. So if the unverified purchases don’t make up the majority of reviews, and they give the product a similar rating to the verified purchases, we won’t devalue the unverified purchase reviews at all.

      • Nero Se says:

        Many thanks for Clearing that up for me , its good to know Review Meta takes this in to account , I wish there was more awareness out there for Amazon buyers to use your site.

        I still think amazon (not RM made a major mistake in allowing fsuch magnitutde of fake reviews, the only action they have taken seems to have been against genuine reviewers like myself . there must be countless others like me.

  3. K.G. says:

    Dear ReviewMeta team,

    Can you please advise whether the Amazon Vine review process is taken into account when classifying reviews as unverified?

    If Vine reviews form a subset of the unverified reviews, is it possible to identify what percentage of the unverified reviews stem from the Vine review process?

    Thank you
    Regards

    K.G.

    • We actually consider Vine reviews to be verified even though Amazon does not. The reason is that we know they received the product, and the unverified test is really looking at reviews that cannot be verified in any way. The report does not show the percentage of vine reviews, but that information can be available upon request.

      • K.G. says:

        Thank you for your response. Please provide me an email address to which I can send specific feedback. I have screen shots from reviewmeta and from Amazon.co.uk which I believe you will be interested in. It may not be prudent to put this level of information up through this blog. I am happy to share the information with you offline. Thank you, regards, K.G.

Leave a Reply