If you're a member of the American College of Physicians and want to learn more about LLMs and their implications for medicine, we put together a webinar!

link: www.acponline.org/clinical-inf...

Based on the psychological literature from my colleagues in the clinical reasoning field, we developed the Management Reason Empirical Key Features scale (MXEKF), a construct that measures many of the important context-specific domains in management reasoning.

What about guidelines? Clinical guidelines are incredibly important -- for many medical conditions (like heart failure), guideline-concordant care has mortality benefits. In fact, we used guidelines from @nicecomms.bsky.social and @bmj.com Best Practice for AMIE's Mx Agent!

We are excited to share our latest results on AMIE, Google Research and Google DeepMind's conversational medical AI. TL;DR – in a blinded OSCE, AMIE was equivalent or better at diagnosis – but also at making management decisions. This is a big deal!

🧵⬇️

So how could we POSSIBLY study this in humans using LLMs???

First, we selected real cases that did not have any right answers. This is easier than TV would have you believe, since actual medicine is replete with shades of grey.

Imagine that you are admitted to the hospital for gallstone pancreatitis. A CT scan picks up a suspicious lung nodule that could be cancer. Should you biopsy it immediately? Wait a few weeks? Perform serial imaging to make sure it doesn't grow? Or do nothing at all?

There's a reason that physicians love Sherlock Holmes – diagnosis is effectively a whodunnit. A single disease causes a patient's symptoms, and doctors can discover this with various types of investigations.

First, the main result: GPT-4 DID improve physician medical management in complex cases with no right answers (though not above the performance of the LLM by itself).

link: www.nature.com/articles/s41...

Our HMS Digital Education and AI conference is back! We're having a prompt competition this year -- how do you use AI to enhance your patient care, home life, or what you do for fun? So sign up!

Competition: forms.gle/YwEHkwk884sZ...

Registration: cmecatalog.hms.harvard.edu/digital-educ...