Thanks! :)

This work is with a team of medical and technology experts I am lucky to collaborate with: Zehavi Kugler, Lior Hayat, Tamar Brufman, Ran Ilan Ber, Keren Leibovich, Tom Beer, Ilan Frank, Ran Shaul (KHealth) Caroline Goldzweig, and Joshua Pevnick (Cedars-Sinai Medical Center).

Natural follow-up questions: causal impacts of AI on decision quality, speed, and outcomes. Best ways to combine human and algorithmic expertise. Trust, over/under reliance, continued learning. Lots of more work to do!

Physicians had to scroll to see recommendations, and we don't know if they did. We didn't study AI impacts. Still, results show well-calibrated models can do very well - more than the average doctor - on
common symptoms that account for the bulk of virtual urgent care cases.

AI excelled at guideline adherence (e.g., no antibiotics for viral URI) & incorporating EHR data (e.g., noticing recurrent UTI history that physicians missed). Physicians did better when patient narratives evolved or when visual findings were important.

Overall scores were equal in 68% of cases, better for AI in 21%, and better for physicians in 11% (these are actual physician decisions in urgent care cases with common symptoms - not medschool exams or vignettes)

AI recommendations were rated optimal in 77% of cases vs 67% for physicians, and potentially harmful in 2.8% vs 4.6% for physicians.

Experienced physician adjudicators viewed full case info and independently rated AI recommendations & physician decisions on a 4-point scale. If two independent adjudicators disagreed, a third was brought in, but we didn't force consensus and weighted final ratings equally.

We sampled 461 adult visits from one month in 2024 with respiratory, urinary, vaginal, eye, & dental symptoms - which in previous work showed had high AI-clinician diagnostic concordance; symptoms represent 2/3 of clinic volume. www.mcpdigitalhealth.org/article/S294...

The AI, commercially developed by KHealth, is an ensemble of discriminative models trained and tested on clinical EHR + med guidelines. To be reliable, the model withholds recommendations when confidence is low (about 1/5 of cases).

The setting is CS-Connect, a telemedicine clinic where AI first automates patient intake via structured chat. AI predicts the differential diagnoses and recommends Rx, labs & referrals. Physicians then conduct a video visit.

Agree! While the focus was AI accuracy, this study is from a clinic where (calibrated, discriminative, ensemble) models already support physician work by expediting intake and providing (when confident) diagnostic predictions and care recommendations. And we plan to study collaboration impacts next!

ishaniganguli.bsky.social @davidmcutler.bsky.social Ateev Mehrotra et al.