Reckon we still have a while before ChatGPT replaces genetic counselors

Of course EvAgg isn't intended to be used as a standalone tool, but as a support for variant scientists - so the key question is whether it helps them. And yes, it does, in a whole variety of ways; users also *really* liked using it, although they were (understandably) still wary of its accuracy.

Overall, EvAgg was *superb* at finding relevant papers in the provided set, with a recall of 0.97, but with a precision of 0.83. It also does a pretty good job of extracting relevant content, though there's clearly plenty of further room for improvement here.

It's worth noting that benchmarking against human performance for this kind of complex task is... non-trivial. Interestingly, later assessment of points where EvAgg and the human-curated truth set disagreed suggested that the AI was probably more correct about half the time.

That's cool in theory, but does it actually work? To test that, the team chose 45 disease-associated genes and manually curated a bunch of literature on those genes. Kudos to the variant science team, including Lynn Pais, Sam Bryen, Emily O'Heir, and Chrissy Austin-Tse.

From a variant curator's perspective, the key thing is that EvAgg can take a gene symbol as input, extract a bunch of pertinent information from multiple papers, and then deliver a handy table summarising that information.

Our amazing Microsoft colleagues, including Hope Twede and Ashley Conard, developed the Evidence Aggregator (EvAgg), which leverages OpenAI tools including GPT-4o to select relevant papers, extract useful information, and then provide an accessible summary about a query variant/gene.

Excited to be tutoring at the Leena Peltonen School of Human Genetics on July 27-31, alongside a stellar crew. If you’re a late-stage or recently graduated PhD student this is an awesome opportunity to get 1:1 time with faculty at the cutting edge of genomics.

Apply by March 7th at lpshg.com

GPT-4 gets it wrong, but o1-preview has no trouble.