A recent study compared the responses of physicians and an artificial intelligence chatbot, ChatGPT, on a public social media forum.

The study found that the chatbot’s responses were preferred over physicians’ responses and were rated significantly higher for both quality and empathy. However, it is important to note that the chatbot’s responses are designed to be “chatty,” and its almost instantaneous generation of information can place the physician at a significant functional disadvantage in the real world. Additionally, while the chatbot’s responses may be perceived as empathetic, they may not be clinically relevant or actionable. The study’s findings suggest that AI has enormous potential to improve patient outcomes and enhance the patient experience, but further research is needed to explore the limitations and strengths of both physicians and AI chatbots in healthcare.

The recent study comparing the responses of physicians to those of an artificial intelligence chatbot, ChatGPT, on a public social media forum, raises interesting questions about the use of technology in healthcare. The study found that the chatbot’s responses were preferred over physicians’ responses and were rated significantly higher for both quality and empathy.

Let’s take a look at how this might play out in the real world.

While this is an interesting finding, it is important to note that the responses generated by the chatbot are designed to be “chatty” and the almost ‘instantaneous” AI generation of information can place the physician at a significant functional disadvantage—at least a perceptual one.

Physicians are hardwired to provide factual information, often within specific clinical and legal demands. They are pressed for time and need to prioritize information that is clinically relevant and actionable. In contrast, the chatbot’s responses are designed to mimic natural language and appear more conversational. This could be perceived as empathetic, but it may not be clinically relevant or actionable.

Another factor that could be contributing to the utility to the chatbot’s responses is the speed of GPT’s creation. The chatbot can generate a relatively well-informed clinical reply in long form almost instantly, while a similar response from a physician in terms of length and depth would be impractical, if not impossible, from both a time and financial perspective. Physicians are typically pressed for time and must balance the need to provide quality care with the realities of a busy schedule. Chatbots, on the other hand, can be programmed to access a vast amount of medical knowledge, and their responses can be generated in real-time, making them a valuable resource for patients seeking quick, but and robust answers to their health questions.

Nevertheless, the study’s observations, whether valid or a bit contrived, provide an eye-opening moment for clinicians, patients, and all those who have been touched by the power of GPT. The use of artificial intelligence in healthcare has enormous potential to improve patient outcomes, reduce clinician burnout, and enhance the patient experience. However, it is important to use these tools appropriately and ensure that they complement the expertise of clinicians. Let’s keep the word ‘replace’ out of the discussion for now, but the potentiality certainly looms on the horizon for many parts, if not all of a clinical dialogue.

JAMA Internal Medicine, April 2023.

The JAMA study comparing physician and artificial intelligence chatbot responses to patient questions is an interesting first step in exploring the role of AI in healthcare. While the study’s findings suggest that chatbots can generate high-quality and empathetic responses, it is important to consider the limitations of the study and the unique strengths and limitations of both physicians and AI chatbots. The observations from this study are just the beginning, and further research is needed to explore the potential of AI in healthcare.