Online reviews of health products 'are misleading'
"Don't believe online reviews of health products, they're 'skewed'," the Mail Online reports.
A psychologist compared online reviews of three medical products with results from clinical trials, and found the reviews are skewed towards the positive.
The author of the study, Dr Micheál de Barra, wanted to look into whether people who have had good outcomes from treatments are more likely to go online and give positive reviews than people who have had average or poor outcomes. As such, the product reviews provided by online retailers may be distorted.
The author looked at Amazon.com – the US version of the site – and analysed two cholesterol-reducing products and one weight loss treatment.
In general, he found the extent of cholesterol reduction or weight loss reported by online reviewers was substantially greater than that demonstrated in randomised controlled trials, a more reliable source of evidence on effectiveness.
The research highlights an underlying issue with online reviews, whether they are for health products, films or books. Online reviews are arguably subject to a type of reporting bias in that they are written by people who take the time to write them.
This means it's far more likely that these reviews are written by people who have very strong views, either positive or negative, about a product than people who would score it three out of five stars.
Useful impartial online resources for assessing the effectiveness of medical products and treatments include NHS Evidence, the TRIP Database and the Cochrane Library.
Where did the story come from?
The study was authored by a single researcher from the University of Aberdeen and published in the peer-reviewed journal, Social Science and Medicine.
No sources of financial support are mentioned. The author reports support and advice from four named individuals, but otherwise declares no conflict of interest.
The UK media's coverage of this study is generally representative, but it would be unfair to single out Amazon or the products discussed as having particularly misleading reviews.
The site and the products just happen to be the ones the author looked at. It's likely other products on other sites have equally skewed reviews.
What kind of research was this?
This study aimed to look into the potential issue of biased reporting by online retailers around the effects of medical treatments.
The author reports how people "often hold unduly positive expectations about the outcomes of medicines and other healthcare products".
He suggests the reason for this could be that people who have had a positive outcome from a treatment are more likely to be vocal about this, compared with people who've had poor to average outcomes.
This means the information available to others through customer reviews on retailers' websites is likely to be distorted in favour of the product.
The author focused on several questions:
- Whether there is biased online reporting around medical treatments compared with the evidence from clinical trials.
- Whether there is consistent reporting bias across all treatments targeting the same health problem.
- Whether those with poor or average outcomes are less likely to report their experience than those who had a positive experience.
What did the researcher do?
The author aimed to see whether the reviews of medical products published by the international online retailer Amazon.com are consistent with outcomes of the same products reported in clinical trials.
If there is no reporting bias, the average outcomes should be more or less the same.
He found three medical products available that met the following criteria:
- they were genuine medical products
- they had more than 300 online reviews
- the reviews contained specific information about the reviewer's health
- there was high-quality evidence (such as randomised controlled trials) of the products available to check the true clinical effects
The three products that met these criteria were two cholesterol treatments – Benecol Smart Chews Caramels (not available in the UK) and Nature Made CholestOff (a herbal supplement available over the counter) – and a weight loss drug designed to reduce the amount of fat the body takes from food, called Orlistat (also available over the counter under the supervision of a pharmacist).
As the author says, it's likely that many other treatments would also have met these criteria, but these were the first three identified.
He included a total of 908 reviews of the two cholesterol products where the reviewers also provided information on changes in blood cholesterol levels.
These were compared with the size of the effects reported in a systematic review of clinical trials.
There were 767 reviews of Orlistat included – a specific weight change was reported in about a third of these.
A recent systematic review was identified for comparison, which included two clinical trials of this drug.
What did he find?
For the cholesterol products, the author found the blood cholesterol reductions reported by online reviewers were significantly greater than those reported in clinical trials.
For example, Benecol users reported a 45mg/dl reduction in cholesterol, compared with a range of 9.28-24mg/dl reduction reported in trials.
Similarly, the weight loss reported by Orlistat reviewers was significantly greater than in the clinical trial participants (specific kg loss not reported in the study).
The author calculated that the "reputational distortion" (the perceived benefit) for the three products was three to six times greater than the actual size of the benefit gained from the treatment.
When looking at the one- to five-star grading system, on average, reviews that were shared had 0.55 more positive stars than those that weren't shared.
What did the researcher conclude?
The author concluded that, "People with good treatment outcomes are more inclined to share information about their treatment, which distorts the information available to others.
"People who rely on word of mouth reputation, electronic or real life, are likely to develop unduly positive expectations."
Conclusions
This unique study suggests that, in general, online medical product reviews may give a distorted and enhanced perception of the effectiveness of the product compared with that actually demonstrated in randomised controlled trials.
The author discusses potential theories around this. For example, it may reflect the fact people are more likely to post a review if they found something good than if the benefit they found was not that remarkable or there was no benefit at all.
He also suggests people may not wish to dwell on prior periods of ill health, whereas a positive recovery is something they may want to share with others.
People who remain in poorer health could also have lower mood and be less inclined to engage in sharing information about their health.
But, as the author acknowledges, in the case of raised cholesterol this theory doesn't provide the whole answer because the condition doesn't cause as many obvious health problems.
However, it's important to note a few points:
- Fraudulent or fake reviews are rare and were not found in the data reviewed in this study. The benefits reported may have been greater than those found in clinical trials, but they were still reporting the same effect: lower cholesterol or weight loss.
- Around 90-100% of reviews are thought to be reliable, or at least written in "good faith" and not in an attempt to mislead people.
- People who achieve big results with certain products may be the exception rather than the rule, and most people may be more likely to see average results.
- We don't have any information about other lifestyle changes or medical treatments people may also have been using. For example, those people achieving greater weight loss with Orlistat may also have been making diet and physical activity changes.
- This study has not singled out Amazon or these three specific products. As the author says, he could have looked at many other products, but these just happened to be the first three that met the criteria. There are many other general or brand-specific online retailers that provide customer reviews of their products, so the focus should not just be on Amazon.
While not as user-friendly as Amazon, sites such as NHS Evidence, TRIP Database and the Cochrane Library provide impartial and unbiased assessments of medical products and treatments.
When it comes to lowering your cholesterol or trying to lose weight, it's advised that you follow official diet and physical activity guidelines.
