Is ChatGPT in Your Doctor’s Inbox?

May 3, 2023 — What happens when a chatbot slips into your doctor’s direct messages? Depending on who you ask, it might improve outcomes. On the other hand, it might raise a few red flags.

The fallout from the COVID-19 pandemic has been far-reaching, especially when it comes to the frustration over the inability to reach a doctor for an appointment, let alone get answers to health questions. And with the rise of telehealth and a substantial increase in electronic patient messages over the past 3 years, inboxes are filling fast at the same time that doctor burnout is on the rise.

The old adage that timing is everything applies, especially since technological advances in artificial intelligence, or AI, have been rapidly gaining speed over the past year. The solution to overfilled inboxes and delayed responses may lie with the AI-powered ChatGPT, which was shown to substantially improve the quality and tone of responses to patient questions, according to study findings published in JAMA Internal Medicine. 

“There are millions of people out there who can’t get answers to the questions that they have, and so they post them on public social media forums like Reddit Ask Docs and hope that sometime, somewhere, an anonymous doctor will reply and give them the advice that they are looking for,” said John Ayers, PhD, lead study author and computational epidemiologist at the Qualcomm Institute at the University of California-San Diego.

“AI-assisted messaging means that doctors spend less time worried about verb conjugation and more time worried about medicine,” he said. 

r/Askdocs vs. Ask Your Doctor

Ayers is referring to the Reddit subforum r/Askdocs, a platform devoted to providing patients with answers to their most pressing medical and health questions with guaranteed anonymity. The forum has 450,000 members, and at least 1,500 are actively online at any given time.

For the study, he and his colleagues randomly selected 195 Reddit exchanges (consisting of unique patient questions and doctor answers) from last October’s forums, and then fed each full text question into a fresh chatbot session (meaning that it was free of any prior questions that could bias the results). The question, doctor response, and chatbot response were then stripped of any information that might indicate who (or what) was answering the question – and subsequently reviewed by a team of three licensed health care professionals. 

“Our early study shows surprising results,” said Ayers, pointing to findings that showed that health care professionals overwhelmingly preferred chatbot-generated responses over the physician responses 4 to 1. 

The reasons for the preference were simple: better quantity, quality, and empathy. Not only were the chatbot responses significantly longer (mean 211 words to 52 words) than doctors,  but the proportion of doctor responses that were considered “less than acceptable” in quality was over 10-fold higher than the chatbot (which were mostly “better than good”). And compared to doctors’ answers, chatbot responses were more often rated significantly higher in terms of bedside manner, resulting in a 9.8-fold greater prevalence of “empathetic” or “very empathetic” ratings.

A World of Possibilities

The past decade has demonstrated that there is a world of possibilities for AI applications, from creating mundane virtual taskmasters (like Apple’s Siri or Amazon’s Alexa) to redressing inaccuracies in histories of past civilizations.

In health care, AI/machine learning models are being integrated into diagnosis and data analysis, e.g., to speed up X-ray, computed tomography, and magnetic resonance imaging analysis or help researchers and clinicians collate and sift through reams of genetic and other types of data to learn more about the connections between diseases and fuel discovery.

“The reason why this is a timely issue now is that the release of ChatGPT has made AI finally accessible for millions of physicians,” said Bertalan Meskó MD, PhD, director of The Medical Futurist Institute. “What we need now is not better technologies, but preparing the health care workforce for using such technologies.”

Meskó believes that an important role for AI lies in automating data-based or repetitive tasks, noting “any technology that improves the doctor-patient relationship has a place in health care,” also highlighting the need for “AI- based solutions that improve their relationship by giving them more time and attention to dedicate to each other.”

The “how” of integration will be key.

“I think that there are definitely opportunities for AI to mitigate issues around physician burnout and give them more time with their patients,” said Kelly Michelson, MD, MPH, director of the Center for Bioethics and Medical Humanities at Northwestern University Feinberg School of Medicine and attending physician at Ann & Robert H. Lurie Children’s Hospital of Chicago. “But there’s a lot of subtle nuances that clinicians consider when they’re interacting with patients that, at least right now, don’t seem to be things that can be translated through algorithms and AI.”

If anything, Michelson said that she would argue that at this stage, AI needs to be an adjunct.

“We need to think carefully about how we incorporate it and not just use it to take over one thing until it’s been better tested, including message response,” she said. 

 Ayers agreed. 

“It’s really just a phase zero study. And it shows that we should now move toward patient-centered studies using these technologies and not just willy-nilly flip the switch.”

The Patient Paradigm

When it comes to the patient side of ChatGPT messaging, several questions come to mind, including relationships with their health care providers.

“Patients want the ease of Google but the confidence that only their own provider may provide in answering,” said Annette Ticoras, MD, a board-certified patient advocate serving the greater Columbus, OH, area. 

“The goal is to ensure that clinicians and patients are exchanging the highest quality information.The messages to patients are only as good as the data that was utilized to give a response,” she said. 

This is especially true with regard to bias.

“AI tends to be kind of generated by existing data, and so if there are biases in existing data, those biases get perpetuated in the output developed by AI,” said Michelson, referring to a concept called “the black box.” 

“The thing about the more complex AI is that oftentimes we can’t discern what’s driving it to make a particular decision,” she said.  “You can’t always figure out whether or not that decision is based on existing inequities in the data or some other underlying issue.”

Still, Michelson is hopeful.  

“We need to be huge patient advocates and make sure that whenever and however AI is incorporated into health care, that we do it in a thoughtful, evidence-based way that doesn’t take away from the essential human component that exists in medicine,” she said.