AI Symptom Checkers: Accuracy, Risks, and When Not to Use Them

What the landmark study found

The most-cited evaluation is the 2015 BMJ audit by Semigran and colleagues, which tested 23 free online symptom checkers against 45 standardized patient vignettes^[1]. Two numbers get mixed up constantly, so read carefully:

The "34%" figure is diagnosis-listed-first accuracy — not triage. Triage advice was appropriate 57% of the time overall, and performance varied by urgency: 80% for emergencies, 55% for non-urgent cases, and 33% for self-care situations^[1]. In other words, checkers were reasonably good at saying "go to the hospital now" but poor at the calm end — and they tended to be over-cautious, sending people to care they didn't need. A 2022 systematic review in npj Digital Medicine reached the same qualitative conclusion: diagnostic accuracy is generally low, triage accuracy moderate and highly variable between tools^[2]. Newer AI-based checkers are improving, but no independent evaluation we could verify shows them approaching clinician-level reliability.

When a symptom checker is useful — and when it is dangerous

A useful pattern: use a checker to prepare for a conversation with a clinician — organizing symptoms, duration and history — not to decide whether you need one. The dangerous pattern is the reverse: using a checker to talk yourself out of seeking care. In the BMJ audit, checkers failed to triage roughly 1 in 5 emergencies appropriately^[1]. No consumer checker should be used to decide whether to skip emergency care.

The contrast worth knowing: NHS 111 online in England is a clinician-governed national triage pathway for people aged 5 and over — it feeds into real services (999, A&E, GPs, pharmacists) with nurse callbacks prioritized by severity^[3]. That is a fundamentally different category from an unregulated chatbot.

AI mental health apps: the evidence, honestly

The research base is small and early. The foundational Woebot study (2017) randomized 70 students to a CBT-based chatbot or an ebook for two weeks and found self-reported reductions in depression and anxiety symptoms in the chatbot group^[4] — promising, but small, short, self-reported, and no substitute for a therapy trial. Wysa has a 2024 peer-reviewed RCT among people with chronic diseases, and received an FDA Breakthrough Device designation in May 2022 — a designation is a fast-track flag, not an approval^[5]. The apps themselves are honest about their limits: Wysa states it "cannot diagnose, treat, monitor or cure" and is not for crisis use.

There is one regulated exception: Rejoyn, cleared by the FDA in April 2024 as the first prescription digital therapeutic — a six-week program used alongside clinician-managed outpatient care for adults with major depressive disorder, prescribed by professionals^[6]. Even Rejoyn is an adjunct, not a replacement. And on privacy: Mozilla's 2022 review found most mental-health apps failed basic privacy and security standards, with data-sharing practices described as "terrible"^[7] — read the privacy policy before typing anything personal.

ChatGPT and medical questions

Chatbots can sound astonishingly medical. ChatGPT performed at or near the passing threshold on USMLE-style exam questions in a 2023 study^[8], and in another study evaluators preferred its answers to physicians' 79% of the time, rating them more empathetic^[9]. Read that again: those studies measured exam answers and perceived warmth, not clinical safety. Meanwhile, a 2023 study found that 55% of GPT-3.5's generated citations — and 18% of GPT-4's — were fabricated, with substantive errors in many of the real ones^[10]. A tool that invents references should never be your source for health decisions. The CDC puts it plainly: treat generative AI as "a drafting and synthesis aid, not an authority," because it "can confidently produce incorrect or fabricated content"^[11]. And legally, chatbots are not medical devices: regulators base that status on the manufacturer's intended purpose, which for ChatGPT is not medicine^[12].

Frequently asked questions