Online symptom checkers' reliability assessed

"The danger of an online diagnosis: Millions of emergencies are MISSED through symptom checker websites," the Mail Online reports. American researchers have assessed the accuracy and reliability of 23 symptom checkers, including the NHS Choices symptom checker.

The researchers were looking at both accuracy of diagnosis and triage. Triage is the process of determining the severity of a condition.

They did this by using a series of symptoms and medical history known to be associated with specific conditions. These types of combination are known as clinical vignettes.

The NHS Choices symptom checker doesn’t offer a diagnosis – it only provides triage advice.

Researchers found that the NHS Choices symptom checker correctly identified emergency and urgent conditions in 87% of cases. But it also incorrectly triaged non-emergency or non-urgent conditions in 80% of cases, which theoretically would result in an unneeded visit to A&E or a call for an ambulance.

While symptom checkers are far from perfect, they are better than just leaving people "Googling" their symptoms, which is estimated to have a 64% success rate in identifying emergency and urgent cases.

Where did the story come from?

The study was carried out by researchers from Harvard Medical School, Brigham and Women’s Hospital, Boston Children’s Hospital and Beth Israel Deaconess Medical Center, all in Boston. It was funded by the US National Institutes of Health.

Researchers involved in this study say they have not been, nor plan to be, involved in the development, evaluation, promotion, or any facet of a Harvard Medical School-related symptom checker and they had no support from any organisation for the submitted work. This includes no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, or other relationships or activities that could appear to have influenced the submitted work.

In the interests of transparency, we should also point out the Behind the Headlines editorial team is employed by the Health & Social Care Information Centre, which is the same NHS organisation that runs the NHS Choices symptom checker.

The study was published in the peer-reviewed British Medical Journal on an open-access basis, so the study is free to read online or download as a PDF.

It was reported by the Mail Online website in the UK. Overall, the Mail reported the story accurately, but the limitations of the study were not fully explained. Its headline "Millions of emergencies are MISSED through symptom checker websites, study warns" is untrue. The study provides no estimate of how many emergency cases were misdiagnosed by symptom checkers around the world.

What kind of research was this?

This was an audit study that aimed to assess the diagnostic and triage accuracy of online symptom checkers (tools that use computer algorithms to help patients with self-diagnosis or self-triage). Triage is the process of determining the priority of patients' treatments, based on the severity of their condition.

With improvements in technology and access to the internet, people are increasingly using the internet to research their health concerns. For example, researchers quote that the NHS Choices website has over 15 million visits per month. That figure was actually based on 2012 data; the figure for 2015 averages around 50 million visits a month.

Although there are many advantages to using symptom checkers, such as providing out-of-hours advice and reducing the burden on GP and A&E departments, they cannot always take the place of face-to-face clinical assessment.

What did the research involve?

Researchers searched for online symptom checkers that were in English, free to access, publicly available, and did not focus on a single type of condition.They used terms like "symptom checker" or "medical diagnosis" to find symptom checkers in Google and Google Scholar, and also searched for any relevant apps in the Apple app store and Google Play.

After searching and sifting, they finally included 23 online symptom checkers for further analysis. They categorised symptom checkers by whether they facilitated self-diagnosis, self-triage, or both. They also categorised them according to the type of organisation they were operated by, the maximum number of diagnoses provided and whether they were based on Schmitt or Thompson nurse triage guidelines. These are decision support protocols commonly used in telephone triage for paediatric and adult consultations, respectively.

To evaluate the diagnosis and triage performance of the symptom checkers, they used 45 standardised patient vignettes. They say they used this method to assess the performance of the symptom checkers, because this method is commonly used by physicians and other clinicians on their diagnostic ability and management decisions. These 45 clinical vignettes were further divided as either "common" or "uncommon" diagnoses. Triage advice was further divided into three groups:

Emergency, which included advice to call an ambulance, go to the emergency department, or see a general practitioner immediately.
Non-emergency, which included advice to call a general practitioner or primary care provider, see a general practitioner or primary care provider, go to an urgent care facility, go to a specialist, or go to a retail clinic.
Self care, which included advice to stay at home or go to a pharmacy.

Each standardised patient vignette was entered into each website or app, and resulting diagnosis and triage advice was recorded.

What were the basic results?

The 23 symptom checkers included in this study were based in the UK, US, the Netherlands and Poland. The 45 standardised patient vignettes used to assess the performance of these symptom checkers included 26 common and 19 uncommon diagnoses.

Performance on diagnosis

Overall, the correct diagnosis was listed first 34% of the time. For different levels of triage, the percentage of the correct diagnosis being listed first is below:

24% (95% confidence interval (CI) 19% to 30%) for emergency evaluations
38% (95% CI 32% to 34%) for non-emergency evaluations
40% (95% CI 34% to 47%) for self care evaluations

Performance on triage advice

The included online symptom checkers correctly gave triage advice 57% of the time. For different levels of triage, the percentage of correct advice is below:

80% (95% CI 75% to 86%) for emergency care evaluations
55% (95% CI 47% to 63%) for non-emergency evaluations
33% (95% CI 26% to 40%) for self care evaluations

How did the researchers interpret the results?

Researchers concluded by saying, "Physicians should be aware that an increasing number of their patients are using new internet-based tools such as symptom checkers and that the diagnosis and triage advice patients receive may often be inaccurate. For patients, our results imply that, in many cases, symptom checkers can give the user a sense of possible diagnoses, but also provide a note of caution, as the tools are frequently wrong and the triage advice overly cautious."

They added, "Symptom checkers may, however, be of value if the alternative is not seeking any advice or simply using an internet search engine. Further evaluations and monitoring of symptom checkers will be important to assess whether they help people learn more and make better decisions about their health."

Conclusion

This audit study showed that online symptom checkers sometimes correctly diagnose and advise people according to their symptoms, but they can be inaccurate. Though the appropriate triage advice was only accurate, on average, 57% of the time, the study found this advice erred on the side of caution, advising people to seek help.

There are several limitations to this study, including:

The study did not include real people, but relied on clinical vignettes to assess the performance of the online symptom checkers. These vignettes included medical terms, which would not necessarily be used by people accessing the sites. Actual patients may sometimes find it difficult to express their symptoms or use different terms. On the other hand, people may refine or add more detail if the advice received was not what was expected.
The study might not have captured all the online symptom checkers available.
The study does not compare the diagnosis and advice of the online symptom checker to the diagnosis and advise that would have been provided by a doctor.

Overall, it is important to note that symptom checkers should be used as an indicator, and not viewed as an alternative to seeking medical advice, especially if you think it may be a medical emergency.

Like symptom checkers themselves, when it comes to assessing a situation, it is always better to err on the side of caution.