TripAdvisor’s Transparency Report Claims 0.6% Fake Reviews. Fakespot Claims 33%. Who is Right?
Before we begin, we want to congratulate TripAdvisor on taking the initiative to be more transparent about their review moderation and fraud detection process and numbers. We think this is something EVERY major review platform should be required to do, and have some thoughts on how platforms could take this several steps further. Be on the lookout for our next post on the topic of required transparency for major review platforms.
Last month, TripAdvisor published their first-ever transparency report. It’s an easy 29 page read with all sorts of numbers, facts, figures and fancy buzzwords. Their stated reason for releasing the report can be found on page 4:
too often we have seen third parties share inaccurate figures about the authenticity of review content and the scale of fraud on our platform
Public outcry about fake reviews is not a new phenomenon, but over the summer, Fakespot claimed that 33% of reviews on TripAdvisor were fake. Yikes.
(For those that aren’t familiar with Fakespot – they are a review analysis site just like ReviewMeta, but also support platforms outside of Amazon – like TripAdvisor. Read about the differences between ReviewMeta and Fakespot here).
Was the timing a coincidence? or was the Transparency Report released as a response to these staggering allegations by Fakespot?
Let’s dive right into the biggest disconnect between Fakespot’s allegations and TripAdvisor’s report – the percentage of fake reviews on TripAdvisor’s platform.
Fakespot: 33% Fake Reviews
TripAdvisor 0.6% Fake Reviews
The first thing we notice is that Fakespot’s number is FIFTY TIMES what TripAdvisor admits. Neither shows a “margin for error”, but one would have to imagine a discrepancy this massive couldn’t possibly be the result of rounding or sampling bias. Something is going on here.
TripAdvisor and Fakespot both have polarized conflicts of interest. On page 4 of the Transparency Report, TripAdvisor admits that they have a vested interest in minimizing the number of fake reviews on their platform:
No one has a greater incentive than TripAdvisor to ensure the reliability of the content on our platform. It is the core of our business because if people don’t find our content reliable and helpful, they won’t keep coming back to our site.
By TripAdvisor’s own admission, their very success is dependent on people perceiving their content as reliable. On the other hand, Fakespot’s success is dependent upon people having the exact opposite perception. If everyone was 100% trusting of every review they read on TripAdvisor, nobody would waste their time with the added chore of looking up review analysis reports on Fakespot.
A counter-argument can be made for each side though. If TripAdvisor claims 0% reviews are fake on their platform, nobody would take their policing efforts seriously, and if Fakespot claimed 100% of reviews on TripAdvisor were fake, people would also start to distrust their analyses.
So who is right? Personally, I doubt that either figure is “right”. As we’ve said a million times before – it’s impossible to determine with 100% accuracy which reviews are “fake” and which are not. Furthermore, it’s not just black and white – there’s a lot of grey area. For example, a review from a restaurant owner’s friend is obviously biased, but what about a negative review on a competitor if that friend actually had a negative experience with said competitor? Is that person prohibited from reviewing any restaurant in the area because their friend owns one?
It’s very likely there are vast methodology differences between TripAdvisor and Fakespot’s fake review detection systems. On page 4, the Transparency Report states:
These third parties do not have access to the key technical data necessary to determine whether or not a review is fraudulent. We do…
While it’s correct that TripAdvisor has access to more data (eg. IP addresses, user agents, email addresses, pre-filtered reviews, etc), there’s still plenty of data that IS disclosed by TripAdvisor which can be used to analyze the reviews. Even the human eye would be able to identify potential fraud, for example, if we see a dozen 5-star reviews posted the same day all with the exact same text.
Furthermore, it’s not the first time that TripAdvisor has rejected the idea that Fakespot could not possibly have enough data to perform a reliable analysis. In direct response to Fakespot’s 33% fake claims, a TripAdvisor representative said:
We totally reject the inaccurate and misleading findings [of Fakespot]. The usefulness and accuracy of the content on TripAdvisor is what has made our site popular to hundreds of millions of consumers. It’s why we fight fake reviews so aggressively.
The claims about fake reviews are based on entirely flawed techniques. Their methods are unreliable for one simple reason: they have no access to the technical data you would need to determine whether or not a review is fake. We do – and we have been using this data for over a decade to track millions of reviews.
If Fakespot’s methods were in any way reliable, we would be using them. We have tested their analysis, with reviews we know to be either genuine or fake, and the results show it to be completely unreliable and inaccurate.
Those are some strong words coming from TripAdvisor. Notice the last part about testing with known genuine or fake reviews. We’ll touch on this a little later.
So let’s compare the two methodologies. (Warning: I’m not a fan of secret formulas or buzzwords, so I get a little snippy here).
From Fakespot’s “How does Fakespot work?” page:
Our proprietary technology analyzes millions of product reviews, looking for suspicious patterns and incentivized reviews.
… K … That literally tells us nothing. Thanks a lot.
After digging around a bit more, I discovered an FAQ page, although it no longer appears to be linked to from anywhere on the site.
Fakespot uses numerous technologies to evaluate the authenticity of reviews. The primary criteria are (1) English language pattern recognition, (2) the profile of the reviewer and (3) correlation with other reviewer data. Our algorithm uses machine learning to constantly improve itself by looking at profile clusters, sentiment analysis and cluster correlation.
This still doesn’t really tell us anything more than some extra buzzwords, but note the “machine learning to constantly improve” part. We’ll touch on this later.
TripAdvisor uses their own share of buzzwords. From page 11:
This technology employs state-of-the art data analytics to review hundreds of pieces of online information that can flag content that doesn’t meet TripAdvisor’s guidelines or that might be fraudulent. It uses network forensics and advanced fraud modeling to map electronic patterns that no human could ever identify.
Fortunately, they go into a little greater detail later on in the report. On page 16 of their report, we see this:
Over the past two decades, TripAdvisor has actively tracked billions of data points across hundreds of millions of reviews, allowing the company to detect clear patterns of what is typical review behavior — and what is not typical. This helps catch fraud. (See illustration below).
Interesting! We see that their methodology is not unlike our own! From the ReviewMeta FAQ (written in 2016), we say almost the same thing:
At ReviewMeta we expect reviews to be natural: they should have predictable patterns that we can easily identify. Since we know what natural reviews look like, it is easy for us to find clues that indicate that reviews might be unnatural.
TripAdvisor describes their review moderation process in better detail between pages 15 and 16. They don’t go into the nitty-gritty or specifics, but they do give readers a general overview of how their system works.
Now let’s revisit that idea of TripAdvisor testing on known genuine vs fake reviews and Fakespot’s claim to use machine learning. To vastly over-simplify Machine Learning, it’s a process in which you feed the computer a set of questions and answers, and the computer builds a model that best predicts the answers based on the questions. (This 9 minute animated video does a really good explaining how it works – I highly recommend watching).
So in order to predict the answers, you need to learn from a set of data that has the answers. To put this into the context of detecting fake reviews, you would need a massive set of known genuine and known fake reviews. TripAdvisor would be much more likely to have this than Fakespot – especially considering the vast majority (73%) of fraudulent reviews submitted to TripAdvisor are identified and blocked before being posted, so they are never visible in the first place (per the Transparency Report, page 17).
That’s not to say that it’s impossible to use Machine Learning with a set of unlabeled data – but TripAdvisor absolutely does have a much richer dataset to help build and refine their algorithms. Again, we’re not saying it’s impossible for Fakespot to run an analysis on the publicly available data, but TripAdvisor does have a point about having access to much more data than Fakespot.
That said, we still have to remember that TripAdvisor admitted in their own transparency report that they have a vested interest in keeping public perception about fakes on their platform to a minimum.
So how does this compare with Amazon?
At ReviewMeta, we only analyze reviews on Amazon, so we can’t weigh in with data on the TripAdvisor debate. But something very interesting is happening on Amazon.
We’re seeing a strikingly similar parallel between the Fakespot vs TripAdvisor and Fakespot vs Amazon debates.
According to an article earlier this year, the estimated percentage of fake reviews on Amazon is:
Fakespot: 30% Fake Reviews
Amazon: <1% Fake Reviews
Again, Fakespot’s estimate is THIRTY TIMES what Amazon is claiming. These numbers are really close to what we see on TripAdvisor. I wonder if there’s a coincidence?
Amazon hasn’t released a transparency report (yet), but here at ReviewMeta, we can throw our own data into the mix.
ReviewMeta: 11% Unnatural Reviews
We land right in the middle. Again, it’s impossible to say who is right, but it’s our (completely biased) belief that Amazon is downplaying the problem, Fakespot is overblowing it, and we’re the closest to reality.
What’s next?
Like we said at the beginning of this post, we applaud TripAdvisor for taking the initiative to release the first-ever Transparency Report. We encourage all other review platforms to do the same and even go beyond just the report.
In a few weeks, be on the lookout for another post about why major review platforms should be required to make their datasets publicly available.
When I travel—and even locally—I consult two review sites that I find to be reliable, particularly if there are more than a handful of reviews. Those two sites are TripAdvisor and Google reviews. Google’s reviews are only lightly moderated, so I’m less confident in their cumulative star ratings unless there are dozens of well-written reviews. On the other hand, I have relied on TripAdvisor for several years to choose lodging, restaurants, etc., and I’ve almost always found their reviews to be refreshingly detailed and accurate.
Anecdotally I came across a couple products recently that were suspicious on Amazon. (Screen protectors for phones) and I ran them through your plugin and every single one of them was thrown out. Most notably due to the fact that out of 20+ reviews not a single one of them was from a verified purchaser.
Amazon could easily detect review bombing in this fashion but refuses to do so. So thanks for doing what you do.