Browser Extensions 

Average Review Weight

Just How Big is the Unnatural Review Problem on Amazon?

May 3rd, 2018

If you’ve been paying attention to the news recently, you may have noticed that the Washington Post released an article about the widespread problem of “fake reviews” on Amazon.  Following this report, the issue remained in the news cycle for a few days, even including a 3-minute segment on Good Morning America:

 

 

It’s always interesting to watch these stories unfold and producers scramble to get some content together for their next day airing.  To me it seems pointless – you can’t fully cover the nuanced and complex issue of inauthentic reviews on Amazon in a mere 3 minute morning news segment.  But I suppose that’s how America prefers to consume their information.

The Untold Story

One of the questions I’ve been getting over and over again is: “just how widespread is the fake review problem on Amazon”?  The reason you’re not seeing the answer in the media is because I don’t have it.

But what I do have is data that shows some interesting trends over the last few years.  Below are two graphs showing some trends we’re seeing in Amazon reviews, based on over 60,000,000 reviews that we’ve collected.  First, here are some caveats on the data:

  1. This is only a sample of the data on Amazon.  We don’t know how big the actual population is, and we also aren’t claiming that this sample is random.  ReviewMeta collects data based on the products our visitors run through our site, so if anything, our sample of data may skew towards the more inauthentic side since visitors might be more likely to run suspicious products through our system in the first place.
  2. Our algorithm is only an estimate.  We’re not claiming to be able to identify fake reviews with 100% accuracy, because it’s impossible to do that.
  3. This query was last run on 4/15/2018.  We’ll try to update this post from time to time to help share a full picture of what’s going on.

That said, let’s look at some graphs.  The graphs show the average weight and average rating for all reviews (that ReviewMeta.com has seen) published on Amazon each month.  This does NOT include reviews which appear to have been removed from Amazon.  The date is based on the date listed on the review, not when we discovered the review.

Average Rating

The average rating is an easy calculation and doesn’t require any interpretation or analysis from ReviewMeta.  Keep in mind that again, we only have a sample of the data, and our sample may not be perfectly random, but that doesn’t really explain the change we see in the data over time.

Average Rating

So the average rating hovers around 4.3, then after the incentivized review ban drops but seems to rebound quickly to the 4.3 average.  However, in the summer of 2017, we see the average rating steadily climb, hitting a peak of 4.42 and bouncing around the 4.35 to 4.4 range.  What changed in the summer of 2017?  It’s not known, but data seems to be indicating that something happened.

Average Review Weight

For every review that ReviewMeta processes, we assign a weight, between 0 and 100.  100 means that the review is fully trusted, and 0 means that the review is not trusted and carries no weight in our adjusted rating calculation.  Keep in mind that the algorithm is designed to be pretty aggressive.  Just because a review is devalued slightly, does not mean it is “fake”.

Average Review Weight

So what you’ll notice that the average weight hovers around 70, but when Incentivized reviews were banned, the average weight jumped to about 75 for nearly 7 months.  However, in the summer of 2017, we noticed a sharp downward trend in the average weight, hovering around 55-60 for the last 6 months.  We aren’t exactly sure what is the cause of this, but it appears to happen simultaneously with the boost in average rating.  Coincidence?  Unlikely.

Does this mean that 40% of reviews on Amazon are “fake”?

Absolutely not!  The average review weight is simply a measure of how trustworthy our algorithm finds the reviews.  As stated earlier, our algorithm is fairly aggressive, and there are certainly plenty of false-positives in our data.  It’s really impossible to figure out the prevalence of “fake” reviews on Amazon, but this chart does show some interesting insights that might indicate the problem could be getting worse.

Stay tuned for updates.

We’ll try to update this post once a quarter so we can monitor what’s going on with the overall picture.  We’re also planning some more posts diving into the world of reviews that have been removed from Amazon (“Deleted Reviews” as we call them in our reports.

If you have thoughts on the causes behind these charts, please share in the comments!