Smart glasses boost AI medical scribes from 81% to 98% accuracy

Medical scribes powered by artificial intelligence are getting a major upgrade. New research shows that adding vision capabilities to these documentation tools dramatically improves their accuracy and could save doctors even more time.

A Flinders University study published in npj Digital Medicine tested AI scribes equipped with smart glasses. The vision-enabled system achieved 98% accuracy compared to just 81% for audio-only versions. The difference comes down to visual cues that matter for patient care but get missed when AI can only listen.

How does it work?

Researchers combined Google's Gemini AI model with Ray-Ban Meta smart glasses to create a scribe that could see and hear patient consultations. The system works like this:

• Smart glasses record video and audio during patient meetings
• AI analyzes both visual and spoken information
• The system generates detailed notes including medication details, patient behaviors, and other visual cues
• Doctors review and approve the AI-generated documentation

In testing, 10 clinical pharmacists recorded 110 mock medication interviews. The sessions included more than 100 different medicine containers - tablets, capsules, injections, and creams. The vision-enabled AI captured medication strength and form 97% of the time, while audio-only systems managed just 28%.

Why does it matter?

Healthcare documentation eats up huge amounts of doctor time. AI scribes already help by listening to consultations and generating notes automatically. But medical consultations involve much more than speech.

"A lot of clinically important information is visual," says Bradley Menz, the study's lead author and academic pharmacist at Flinders University. "Important visual cues during consultations include patients' medicine containers, prescriptions and devices, as well as their body language."

Getting medication details right is crucial for patient safety. When doctors can't accurately document what medications patients are taking, at what doses, and in what forms, it creates risks for drug interactions and dosing errors.

The improved accuracy means less time spent editing AI-generated notes. That translates to more time doctors can spend with patients instead of wrestling with paperwork.

The context

AI scribes have gained popularity in healthcare because they address a real pain point. Doctors spend hours each day on documentation, time that could be spent treating patients. Audio-only AI scribes were a good first step, but they missed crucial visual information.

This study suggests the next generation of AI scribes will need to see as well as hear. Associate Professor Ashley Hopkins, the study's senior author, believes this points toward broader clinical applications beyond just documentation.

But the technology isn't ready for widespread use yet. The researchers note several hurdles that need addressing:

• Privacy concerns around video recording in medical settings
• Patient consent processes for visual AI systems
• Data security for video recordings
• Integration with existing hospital workflows
• Need for human oversight and governance

"This is an augmented tool, not a replacement for clinical judgement," Menz emphasizes. "The clinician still needs to review and sign off the document." The AI system can take screenshots of medication packages and provide full transcripts, giving doctors better information to verify the AI's work.