When the doctor’s note from my daughter’s wellness visit to the hospital was posted on our patient portal, I discovered that the doctor’s AI notetaker had erroneously entered ‘alcohol’ as the ointment applied to a dry skin patch on my daughter’s face, when in fact, my response to the doctor’s question of what I applied to my daughter’s face was ‘Aquaphor.’ Not only did the AI notetaker replace Aquaphor with alcohol, but the scenario revealed that the doctor was probably not actively listening to my responses, perhaps because the AI notetaker used for recording had also been inadvertently delegated the task of active listening.
The integration of frontier Artificial Intelligence (AI) systems into healthcare and medical practices may have brought enhanced productivity and efficiency in the delivery of healthcare services; however, the focus of this essay is when the integration of AI into healthcare takes away more than it gives or leads to avoidable errors, like the replacement of “alcohol” with “Aquaphor” in a note.
AI is currently used by hospitals for management, administration, clinical decision-making, diagnosis, predicting patient outcomes, and personalizing treatment plans. Generative AI (GenAI) tools are used for various tasks, including generating clinical notes, recording inpatient visits, and summarizing telehealth transcripts. For example, Microsoft’s Dragon Ambient eXperience (DAX) copilot is a Generative AI tool used in healthcare. It was launched in 2023 and is used by close to 1,000 clinicians in the US, offering automated documentation of patient visits based on voice recordings, among other capabilities. Furthermore, Dax Copilot is integrated with Epic, a widely used electronic health record platform used by over 40% of hospitals in the US.
When emphasis is placed on efficiency optimization in a healthcare context, the question of what is gained and what is lost becomes important. Gen AI systems in healthcare is increasingly mediating how clinicians capture and create patient-related documentation through the diverse capabilities and intuitiveness built into the AI systems. It follows that this integration of Gen AI and other frontier AI systems into healthcare will come with an eventual decline in a clinician’s dexterity and skills necessary for the optimal delivery of health care, an example of which is active listening, which is very important for doctor-patient encounters.
When AI notetakers become commonplace in clinical encounters, listening to the patient almost becomes the same as recording the patient (of course, after consent has been given), and when doctors delegate listening to the patient’s concerns to AI notetakers, the need to truly listen, actively and intently, to the concerns and responses of the patient reduces. When that happens, efficiency and productivity are unintentionally prioritized over quality of interaction, and that is where the challenge lies.
Doctor-patient encounters often involve nuanced, personal, and humane forms of person-to-person communication that can be lost on AI systems. There are ways in which words are said, with certain types of body language and non-verbal cues that will be almost impossible for AI systems to capture or decipher. When we increasingly allow AI systems to mediate such interactions, some things are gained and some things are lost. Efficiency and productivity are gained, while a genuine human-to-human connection is lost. Furthermore, possible overreliance on AI tools in the example with my daughter and I left the clinician in a place where the accuracy of the AI notetaker was not questioned. It doesn’t take a medical degree for anyone to critically question why a person would think of, let alone apply alcohol to the face of a 6-month-old baby, but that sort of concern can only be realized within a human-to-human conversational context.
While using technology certainly has its benefits, it is very important that clinicians themselves take seriously their responsibility to review the summaries and notes produced by the AI tools, and that any errors should be caught at the review phase. My initial response to the error in the doctor’s note was to call out the doctor (and her assistants) for the error and ask how it was possible that such a mistake was not captured and corrected before the note was signed off and posted to our patient portal. But knowing what I know about AI and agentic systems, especially Gen AI and Large Language Models (LLMs), catching such errors is mostly easier said than done.
By design and functionality, Gen AI and LLMs that operate through AI note takers are built to be as convincing as possible, so that whatever is produced is almost believed to be accurate at face value. Gen AI systems have the tendency to sometimes generate and manufacture non-existent information and present it as factually correct to the unsuspecting user. A study conducted by a Boston University professor revealed that most workers who used agentic AI systems were less likely to trust their corrections of the questionable output produced by the AI systems. While the study was conducted in a business work environment, when used day in, day out for several months or years in a clinical setting, clinicians are very likely to be less inclined to doubt the output produced by AI note-takers. Moreover, a skill like active listening is best developed and honed when it is put to constant use, and it is the grit developed from mastering such a skill over time that will most likely come together in developing the thoroughness and confidence needed to double-check the output of a seemingly efficient AI notetaker, especially when the AI tool has been praised for reliability and accuracy by its developers and investors.
Recently, there has been an increase in billion-dollar investments in deploying AI in healthcare, leading to greater interest and adoption of AI across many healthcare systems and exerting subtle pressure on clinicians to adapt to these technologies, some of whom may not even be able to decline. When technologically-driven adaptive skills are prioritized for clinicians to learn, other humane, personal, and non-technical skills will be unintentionally dismissed.
Therefore, rather than emphasizing technological adoption, more effort needs to be directed towards ensuring that the very skills necessary to provide excellent and humane patient care are not neglected. If doctors unknowingly delegate active listening to AI note takers, an overreliance on those AI systems over time will leave clinicians lagging behind in actively listening and eventually dulling foundational human communication skills. Nobody wants the services of a doctor who does not actively listen to their concerns or misses out on important details because the task of actively listening has been delegated to an AI tool. Over time, clinicians may naturally relent in honing their active listening and other humane skills.
To prevent this, a framework of collaboration should be developed. The efficiency and productivity advanced by AI in healthcare are best achieved when the human capabilities of clinicians are improved and supported. Clinicians should be encouraged to engage in activities that help hone human skills of effective listening, effective communication, and human-centered interactions. Efficiency and optimal productivity through AI must be balanced with human connection and humane quality of care, especially because many patients expect that human-to-human connection from their clinicians. Increased productivity should not be engineered at the expense of quality care within doctor-patient interactions.
Blessing T. Adewuyi, PhD is an Instructor at the University of Georgia.