Study: AI outperforms doctors in summarizing health records

A groundbreaking study featured in the journal Nature Medicine showcased the efforts of an international research team to revolutionize how medical records are summarized. The study focused on comparing large language models (LLMs) against medical professionals in synthesizing vast amounts of data from electronic health records (EHRs), a critical yet time-consuming task in healthcare.

Why does it matter?

Clinical documentation is a critical component of healthcare, involving the meticulous recording of patient histories, diagnostic tests, and treatments. Despite its importance, this process is notably time-consuming and prone to errors, which can have significant implications for patient care.

The shift to EHRs — while beneficial in many ways — has also increased the burden of documentation, with clinicians and nurses spending a substantial portion of their time on this task. This has led to increased stress, burnout, and dissatisfaction among healthcare providers, ultimately affecting patient outcomes.

LLMs present a promising solution for streamlining the summarization of clinical data. However, despite this potential, their effectiveness in clinical settings has not been thoroughly investigated until this study. The research explored the performance of eight LLMs across four types of clinical summarization tasks, including patient inquiries, radiology reports, doctor-patient dialogues, and progress notes.

Key findings

The findings were impressive:

• 45% of the LLM summaries matched the quality of those from medical experts, while 36% were considered superior.
• LLM summaries outperformed expert summaries in being more concise, correct, and complete.

The study highlighted the importance of prompt engineering, or fine-tuning the input prompts, to enhance the models' performance.

For instance, while LLMs excelled in most areas, they fell short in summarizing radiology reports with the same level of conciseness as medical experts. This discrepancy was attributed to the vagueness of the input prompts, suggesting that more precise prompts and additional checks could further refine the summaries' accuracy.

Methodology

The study employed two main types of LLMs: autoregressive and seq2seq models, each suitable for different summarization tasks. It then assessed these models using natural language processing metrics and a clinical reader study, where ten physicians evaluated the models' summaries against those produced by medical experts in terms of conciseness, correctness, and completeness.

The bottom line

The study concludes that LLMs are not only capable of summarizing patient health records as effectively as, if not better than, medical professionals but also offer a promising avenue for improving clinical documentation.

By saving time and reducing the potential for error, LLMs could significantly enhance patient care and alleviate the documentation burden on healthcare providers.