Browser Extensions 

Average Review Weight

Just How Big is the Unnatural Review Problem on Amazon?

May 3rd, 2018

If you’ve been paying attention to the news recently, you may have noticed that the Washington Post released an article about the widespread problem of “fake reviews” on Amazon.  Following this report, the issue remained in the news cycle for a few days, even including a 3-minute segment on Good Morning America:

 

 

It’s always interesting to watch these stories unfold and producers scramble to get some content together for their next day airing.  To me it seems pointless – you can’t fully cover the nuanced and complex issue of inauthentic reviews on Amazon in a mere 3 minute morning news segment.  But I suppose that’s how America prefers to consume their information.

The Untold Story

One of the questions I’ve been getting over and over again is: “just how widespread is the fake review problem on Amazon”?  The reason you’re not seeing the answer in the media is because I don’t have it.

But what I do have is data that shows some interesting trends over the last few years.  Below are two graphs showing some trends we’re seeing in Amazon reviews, based on over 60,000,000 reviews that we’ve collected.  First, here are some caveats on the data:

  1. This is only a sample of the data on Amazon.  We don’t know how big the actual population is, and we also aren’t claiming that this sample is random.  ReviewMeta collects data based on the products our visitors run through our site, so if anything, our sample of data may skew towards the more inauthentic side since visitors might be more likely to run suspicious products through our system in the first place.
  2. Our algorithm is only an estimate.  We’re not claiming to be able to identify fake reviews with 100% accuracy, because it’s impossible to do that.
  3. This query was last run on 4/15/2018.  We’ll try to update this post from time to time to help share a full picture of what’s going on.

That said, let’s look at some graphs.  The graphs show the average weight and average rating for all reviews (that ReviewMeta.com has seen) published on Amazon each month.  This does NOT include reviews which appear to have been removed from Amazon.  The date is based on the date listed on the review, not when we discovered the review.

Average Rating

The average rating is an easy calculation and doesn’t require any interpretation or analysis from ReviewMeta.  Keep in mind that again, we only have a sample of the data, and our sample may not be perfectly random, but that doesn’t really explain the change we see in the data over time.

Average Rating

So the average rating hovers around 4.3, then after the incentivized review ban drops but seems to rebound quickly to the 4.3 average.  However, in the summer of 2017, we see the average rating steadily climb, hitting a peak of 4.42 and bouncing around the 4.35 to 4.4 range.  What changed in the summer of 2017?  It’s not known, but data seems to be indicating that something happened.

Average Review Weight

For every review that ReviewMeta processes, we assign a weight, between 0 and 100.  100 means that the review is fully trusted, and 0 means that the review is not trusted and carries no weight in our adjusted rating calculation.  Keep in mind that the algorithm is designed to be pretty aggressive.  Just because a review is devalued slightly, does not mean it is “fake”.

Average Review Weight

So what you’ll notice that the average weight hovers around 70, but when Incentivized reviews were banned, the average weight jumped to about 75 for nearly 7 months.  However, in the summer of 2017, we noticed a sharp downward trend in the average weight, hovering around 55-60 for the last 6 months.  We aren’t exactly sure what is the cause of this, but it appears to happen simultaneously with the boost in average rating.  Coincidence?  Unlikely.

Does this mean that 40% of reviews on Amazon are “fake”?

Absolutely not!  The average review weight is simply a measure of how trustworthy our algorithm finds the reviews.  As stated earlier, our algorithm is fairly aggressive, and there are certainly plenty of false-positives in our data.  It’s really impossible to figure out the prevalence of “fake” reviews on Amazon, but this chart does show some interesting insights that might indicate the problem could be getting worse.

Stay tuned for updates.

We’ll try to update this post once a quarter so we can monitor what’s going on with the overall picture.  We’re also planning some more posts diving into the world of reviews that have been removed from Amazon (“Deleted Reviews” as we call them in our reports.

If you have thoughts on the causes behind these charts, please share in the comments!

 

 

 


  • Eitan Yaffe

    An interesting estimation for the “size” of the phenomenon can come out plotting the distribution of both the falsity-scores of the reviewers and the falsity-scores of the products (instead of just the mean). I expect both distribs to be bimodal, making them easier to interpret. As you said, you are aggressive, but that will only shift the distribs. Anyway it will allow us to judge ourselves how big is the phenomenon. Keep it up!

    PS If you could follow those distribs over time that might extend the temporal trends you are already seeing :)

    • That’s a good idea – see how products themselves do. The only challenge that I can think of is that products get reviews over time. Individual reviews are submitted on one date. But there are definitely ways we can look at how products Pass/Fail/Warn over time. Definitely want to keep this updated!

  • Eitan Yaffe

    Impressive work! Regarding “… our sample of data may skew towards the more
    inauthentic side …” – it might be a good idea to add for each query one automatic random query, as a control. That pool of queries can help to quantify the skew, or more probably, show the skew is insignificant. Keep it coming :)

    • Thanks!

      As far as the skewed sample goes, it’s the entire sample of data that we’ve collected from Amazon – so, for example, if there are 200 million reviews on ALL of Amazon, and we have collected 100 million of them (our sample), my theory is that the 100 million that remain on Amazon (not yet seen by ReviewMeta.com) are slight less “unnatural” than the ones that we’ve already collected. Again, no way to test or prove that theory because it’s a theory about data that I don’t have.

      • Eitan Yaffe

        Thank you for your reply, I understand. Yet there is a way to statistically support your claim that the entire DB is only slightly less “unnatural” without going through the entire DB. By using a small control (e.g. 10k reviews instead of 100M) that is picked without the user skew (probably from matching product groups), you can get a quick sense of how skewed (or not skewed) your user-selected data actually is. For example, if user-skew is highly pronounced than even a small control group would generate utterly different distributions for your various plots. If you wished you could even quantify the level of the user-skew but a quick comparison of observed vs. control would probably satisfy you here.

        So a relatively small side analysis like this can help you to extend the scope of your conclusions to the entire DB. Hope this clarifies some and is useful!

        • Ah, I see, so you’re saying that I should randomly sample reviews from Amazon and compare that with my dataset? I see how this could provide some interesting insight but I wonder how “pure” of a random sample I could pull for this analysis.

  • Old Marine

    Well the problem is MUCH deeper than most are aware. I have reported this to Amazon and reported HUNDREDS of attempts by Chinese Manufacturers who are offering myself and thousands of other Amazon shoppers FREE items and often CASH on the side as well.

    To circumvent the Amazon Crackdown on only allowing “Verified Purchases” to be reviewed The Chinese simply contact myself (and the others) by email after they have paid to have all of the top 5000 reviewers at Amazon’s Personal Profiles checked. They then gather the email addresses from everyone who shares that on their profile page or from their other sources from prior purchase info etc.

    OK now with this list they will contact me for instance and I will average 10-25 or sometimes MORE emails every single night. They begin just after 6 PM Pacific Standard Time which is 9 AM in China, The emails are always the same. They provide me a list of assorted products that they need reviewed and given FIVE stars:-) Of course you now get to not only keep this product in exchange for the 5 star reviews, often they offer varied amounts of Cash as well. My average is $20.00-$50.00 per item reviewed. Now if I was offered say 5 products that night just from ONE Chinese Company, I could take in $250.00 CASH per day TAX FREE. And this is only ONE of the many MANY offers rolling every single night…Then you also have to factor in all the other cash revenue that can be generated by then selling all these products on Craigslist and/or eBay. Since these reviews themselves are fake as well you can sell these items as BRAND NEW-Never Opened as well. There is no reason to bother even opening these items after all since the whole “Story/Review” itself is entirely fiction you are simply fabricating for Cash.

    This was what was happening when I was a Vine Reviewer at Amazon and once a Top Reviewer there. It got SO bad that we were SO despised I had my Vine Badge removed from my name I was so embarrassed. It is also why I asked to be removed as well. It got to be downright humiliating, so much so I was willing to give up getting some VERY nice products 100% Free twice per month. This included several BIG ticket (to me at least) items including 2 nice Mid-price Sony Vaio Laptops, a VERY nice $1500.00 Miele Vacuum Cleaner. But it was not that generous for the most part, still, it was VERY tough for me to say I was ready to quit. And for the record we KEPT & Used all we got. We could honestly never afford all these things with out combine salaries so it was like money from Heaven for US at least. But it was corrupt. EVERY single reviewer in Vine had a history of ONLY 5 star items for ANY Vine item reviewed. It was beyond embarrassing to me and I simply could not deal with it ANYMORE.

    So now we have a similar situation with Chinese products FLOODING Amazon. Most are not even worth of being there to fill our Landfills! The lack of quality is SO rampant I honestly cannot understand how Amazon can afford to even allow products to be sold without first passing SOME sort of QA assurance. From the exploding Fireball Hover-boards last Christmas on down, the story is always the same. Horrific stories of electrical products, USB cords, you name it, bursting into flames, self destructing…It has just become another eBay but the unknowing Public seems to just be beginning to see that it IS happening to Amazon. It is ALWAYS their third party sellers who have created this eBay like shower of shoddy products with the same total lack of support. They come and go, change their name and/or product lines name & product colors and it just keeps rollin along. And they are taking the AMAZON Name and credibility down that drain WITH them, little by little, inch by inch. It is sad for me since being badly disable I rely on not ONLY Amazon but on HONESTY in their reviews, that is now just a long ago, distant memory. I am unable to leave my home to shop so I find myself now more & more going elsewhere to shop to places where the review process is NOT tainted and I feel a measure of comfort in making my purchases.

    For the record I have consistently always notified Amazon of these nightly emails. I take the FULL message along with the best possible trace I can run on the sender & forward DIRECTLY To Amazon and to a new department they have set up to attempt combating this mess. Sadly all I get in return is a sad little “form” message that says:
    Hello,
    Thank you for letting us know that the reported email may violate our policies.
    We will investigate and take appropriate action. For privacy reasons, we cannot share the results of our investigation with you.

    I have no honest idea if Amazon IS serious or not about cleaning up this mess. I honestly just do NOT know. If they are I wish they would be more open and be CLEAR about how they are going to put a STOP to it! I literally try to avoid the now and that costs ME money often and surely costs Amazon money. I would ask that EVERYONE who EVER sees anything suspicions do as I do and Contact Amazon at the following email to REPORT it to them directly:review-appeals@amazon.com

    Now for us shoppers I have a guideline that has saved e a TON of money by avoiding ALL products that fail either of, or both of the following tests. If you are at Amazon and see an item you want to buy, copy the product URL at the top of that page. OK-Now go to:
    fakespot.com
    and also go to:
    https://reviewmeta.com/

    OK now PASTE that URL into the Box you see at both of those websites. Hit the ENTER button ans wait only a couple seconds for the results to come back. It will open your eyes in a way that will save you from buying so MUCH useless JUNK…

    I hope the above helps at lease someone out there. It has certainly helped myself and my wife and family. I HOPE it ends up helping Amazon! They NEED to fix this. WE need them to fix this.

  • Fallen12345

    Oh boy, yes. I’ve been the face of the Resistance with regard to the highly improper and offensive shenanigans associated with “Beneath [Contempt]” a/k/a the novel known as “Beneath A Scarlet Sky” since last May and, let me tell you, it boggles the mind how Amazon staff have misbehaved on behalf of itself and its imprint Lake Union.

    I wouldn’t begin to know what kind of separate shenanigans may well have been put in play via PR agency, etc. Plenty of bot and 0% trustworthy reviews, and many instances of verbatim review replication by suspicious users reviewing the same book and other products (none of these have been deleted-addressedx so one wonders how many Amazon staff are tasked with review posts).

    Since start of April, I’ve been busy documenting and defending against ill-effects of misfeasance (hacking of my laptop) by Amazon staff for having chutzpah/gall to relay a complaint to them and then the General Counsel about the abomination that is the book but also brazen displacements of the Top Positive Review in favor of not the first runner-up (the impermissible review by author’s best friend posted on first day book was available via KindleFirst program) but the second runner-up by Phil in Magnolia. The TPR displacement was resolved, but not the rest.[1]

    The Top Critical Review(s) spent no time in the rightful spot, and Amazon has been randomly and temporarily rotating in far-from-TCRs all along – not unnoticed and not without protest. I gather I’m to be grateful that (possibly) someone is too fearful (or has a shred of integrity left not) to delete the Resistance-related posts delivering contemporaneous facts uncovered about author’s perfidy upon each new research discovery.

    [1] Why would there be displacement? Because of what can be found in the comments section of the TPR and TCRs about the Frankenstein book-abomination touted as a “85 to 90 percent true” story when it is no such thing, but insults and misuses and abuses almost every individual and group mentioned in it (and that includes the Acknowledgments section!). Do not misuse or abuse the dead or the topic of the Shoah/Holocaust as though a minor character or scenery in a leisure and entertainment product. Not, you’d think, too difficult to grasp for a rational person or company, and yet …

  • doug_jensen

    I get an average of about one solicitation per day to do a fake review. The “confirmed buyer” work-around seems to be that you do buy the product, review it, and then get your purchase price refunded to PayPal.

    • Yes, seems like this is a common strategy these days. Believe it or not, we actually get some sellers filling out the Contact Us form here on ReviewMeta.com asking for reviews!

  • Old Marine

    Well the problem is MUCH deeper than most are aware. I have reported this to Amazon and reported HUNDREDS of attempts by Chinese Manufacturers who are offering myself and thousands of other Amazon shoppers FREE items and often CASH on the side as well.

    To circumvent the Amazon Crackdown on only allowing “Verified Purchases” to be reviewed The Chinese simply contact myself (and the others) by email after they have paid to have all of the top 5000 reviewers at Amazon’s Personal Profiles checked. They then gather the email addresses from everyone who shares that on their profile page or from their other sources from prior purchase info etc.

    OK now with this list they will contact me for instance and I will average 10-25 or sometimes MORE emails every single night. They begin just after 6 PM Pacific Standard Time which is 9 AM in China, The emails are always the same. They provide me a list of assorted products that they need reviewed and given FIVE stars:-) Of course you now get to not only keep this product in exchange for the 5 star reviews, often they offer varied amounts of Cash as well. My average is $20.00-$50.00 per item reviewed. Now if I was offered say 5 products that night just from ONE Chinese Company, I could take in $250.00 CASH per day TAX FREE. And this is only ONE of the many MANY offers rolling every single night…Then you also have to factor in all the other cash revenue that can be generated by then selling all these products on Craigslist and/or eBay. Since these reviews themselves are fake as well you can sell these items as BRAND NEW-Never Opened as well. There is no reason to bother even opening these items after all since the whole “Story/Review” itself is entirely fiction you are simply fabricating for Cash.

    This was what was happening when I was a Vine Reviewer at Amazon and once a Top Reviewer there. It got SO bad that we were SO despised I had my Vine Badge removed from my name I was so embarrassed. It is also why I asked to be removed as well. It got to be downright humiliating, so much so I was willing to give up getting some VERY nice products 100% Free twice per month. This included several BIG ticket (to me at least) items including 2 nice Mid-price Sony Vaio Laptops, a VERY nice $1500.00 Miele Vacuum Cleaner. But it was not that generous for the most part, still, it was VERY tough for me to say I was ready to quit. And for the record we KEPT & Used all we got. We could honestly never afford all these things with out combine salaries so it was like money from Heaven for US at least. But it was corrupt. EVERY single reviewer in Vine had a history of ONLY 5 star items for ANY Vine item reviewed. It was beyond embarrassing to me and I simply could not deal with it ANYMORE.

    So now we have a similar situation with Chinese products FLOODING Amazon. Most are not even worth of being there to fill our Landfills! The lack of quality is SO rampant I honestly cannot understand how Amazon can afford to even allow products to be sold without first passing SOME sort of QA assurance. From the exploding Fireball Hover-boards last Christmas on down, the story is always the same. Horrific stories of electrical products, USB cords, you name it, bursting into flames, self destructing…It has just become another eBay but the unknowing Public seems to just be beginning to see that it IS happening to Amazon. It is ALWAYS their third party sellers who have created this eBay like shower of shoddy products with the same total lack of support. They come and go, change their name and/or product lines name & product colors and it just keeps rollin along. And they are taking the AMAZON Name and credibility down that drain WITH them, little by little, inch by inch. It is sad for me since being badly disable I rely on not ONLY Amazon but on HONESTY in their reviews, that is now just a long ago, distant memory. I am unable to leave my home to shop so I find myself now more & more going elsewhere to shop to places where the review process is NOT tainted and I feel a measure of comfort in making my purchases.

    For the record I have consistently always notified Amazon of these nightly emails. I take the FULL message along with the best possible trace I can run on the sender & forward DIRECTLY To Amazon and to a new department they have set up to attempt combating this mess. Sadly all I get in return is a sad little “form” message that says:
    Hello,
    Thank you for letting us know that the reported email may violate our policies.
    We will investigate and take appropriate action. For privacy reasons, we cannot share the results of our investigation with you.

    I have no honest idea if Amazon IS serious or not about cleaning up this mess. I honestly just do NOT know. If they are I wish they would be more open and be CLEAR about how they are going to put a STOP to it! I literally try to avoid the now and that costs ME money often and surely costs Amazon money. I would ask that EVERYONE who EVER sees anything suspicions do as I do and Contact Amazon at the following email to REPORT it to them directly:review-appeals@amazon.com

    Now for us shoppers I have a guideline that has saved e a TON of money by avoiding ALL products that fail either of, or both of the following tests. If you are at Amazon and see an item you want to buy, copy the product URL at the top of that page. OK-Now go to:
    fakespot.com
    and also go to:
    https://reviewmeta.com/

    OK now PASTE that URL into the Box you see at both of those websites. Hit the ENTER button ans wait only a couple seconds for the results to come back. It will open your eyes in a way that will save you from buying so MUCH useless JUNK…

    I hope the above helps at lease someone out there. It has certainly helped myself and my wife and family. I HOPE it ends up helping Amazon! They NEED to fix this. WE need them to fix this.

  • Hey Andrea-

    Thanks for commenting. The data in this post is specifically from reviews on Amazon.com, however the data on Amazon.co.uk isn’t too far off (although we have a MUCH smaller sample from there).

    Amazon has a tough problem on their hands, and consumers are growingly increasingly impatient with them (rightfully so!). I definitely think there’s more they could be doing and that’s one of the reasons I started ReviewMeta. Hopefully as we grow and help identify problematic brands and reviews, it will help Amazon shape Amazon’s policies in the future.

    • andrea

      You mention Facebook in another reply; that is a fine example… I have learnt to treat ALL the sponsored content I see as spam, regardless of who they are (pretending to be) representing.
      I can’t help wondering if Amazon are willing to go down the same route. Considering that I am quite sure Bezos & Co. are not stupid, what they are (not) doing must make (financial?) sense to them.

  • Hey Lawrence –

    Thanks for your comment. ReviewMeta is definitely not a replacement to actually reading the reviews. We’re not perfect and that’s why we show all our data so you can make an informed decision. However, I think that the reports have some validity to them. Check out this example author – lots of books with unnatural looking reviews: https://reviewmeta.com/brand/shon-brooks

    • Lawrence Ambrose

      Well, to be clear, I think you’re doing an outstanding service. I use you all the time, and we agree far more often than not. No question there’s some – more than some – validity to your reports. Your algorithms strike me as excellent for the most part. The sheer number of artificially inflated sales/reviews in ebooks blows my mind. Here’s one example: some dude launches some books with terrible covers, formatting, and barely literate English and makes over 10K in three months. I bet you know whom I talking about. :)

  • Hey OnlineBiz22-

    I totally agree with you here. There’s that catch-22 where if you don’t have reviews, you won’t sell your product. If you don’t sell your product, you won’t get reviews. I believe the solution is in some sort of in-house editorial team or vast expansion and improvement of the Vine program like you suggested. Make it a requirement to send out 20 units to vine reviewers in order to list a new product on Amazon. Or start a “secret shopper” program to ensure that sellers aren’t shipping out a different product to the reviewers.

    • Lawrence Ambrose

      Amazon has the money to hire editorial reviewers who would check out stuff and review based on merit. Yes, that would require some expense, but when you consider all the money Amazon spends pointlessly on its control-freak stuff (such as reviewing your covers, which no other online book broker does) they ought to be able to substitute programs that actually help people. Controversies would no doubt stem from authors claiming unfairness, but even if it wasn’t utterly fair, reviews could still be helpful to sellers overall, in my opinion.

  • Hey MakeAmericaAgain! –

    This is a great question that everyone should be asking about ALL companies. Currently ReviewMeta.com generates revenue from the banner ads you see around the site. We don’t get a commission from Amazon or receive payments from brands in any way. Hopefully you’ll see that we’re not here to extort anyone, and that our algorithm isn’t influenced by money in any way.

    • MakeAmericaAgain!

      Thank you for the clarification. Yes, it should always be asked – especially for free services, regardless of the perceived value (and I see your value as very high)!

      • Absolutely. For example, one thing we’re seeing in the supplement industry is a lot of fake “unbiased reviews” sites popping up and recommending really scammy, overpriced products. These bloggers claim to have tried thousands of supplements, and are recommending only the most effective. However, their most recommended supplements also earn them a fat commission, sometimes up to 45%!

        So this is a great question to be asking ANYONE offering a free service. Look at Facebook – that’s a free service and we’re just now beginning to collectively realize the cost.

        • MakeAmericaAgain!

          Labdoor.com is a great site for supplement ratings.

  • Bob Stuart

    I wonder if Amazon’s “competitors” are actually subsidiaries. A few years ago Hong Kong had some good deals, but I now have received a steady stream of factory rejects from four different companies that won’t accept low-star reviews for “technicalities” and make refunds a pennies-per-hour proposition.

  • Mike

    You guys are the best.

    But it would be nice if I could search ReviewMeta for more reputable reviewed items instead of just browse.

    • Thanks Mike!

      I would also like that feature. We have category pages, but it’s really tough to line up products on an apples-to-apples basis. I’ll try and revisit this idea soon to see what I can come up with it.

  • MakeAmericaAgain!

    Interesting – I bought USB cables based on their high review score. They would not charge my iPad. When I just went back to the page, I see a rating of 2.5 stars based on 15 reviews. When I open ReviewMeta, it shows the original score of 4.9 stars from 1,749 reviews and an adjusted score of 2.5 stars from 250 reviews. When I refresh the rating, I see an original score of 4.9 stars from 120 reviews.

    https://www.amazon.com/gp/product/B07923G947
    https://reviewmeta.com/amazon/B07923G947

    • Hi MakeAmericaAgain-

      This is likely due to how Amazon groups reviews for different product versions into one pool. If you look at the report, just below the product image, it shows “223 Alternate Versions” – sometimes Amazon splits apart the reviews for different versions and it’s tough for us to keep it updated. As it stands, right now the only way to fix this on our end is for someone to manually contact us and we can reset the grouping. I’m looking into better ways of keeping it updated though.