13/13 Thanks to all the amazing collaborators: Martina Occhetta, Nik Branson, Magdalena Huebner, Victor Curean, Will Dee, Will Connell, Alex Hawkins-Hooker, Pui Chung, Yasha Ektefaie, Amaya Gallagher-Syed (@amayags.bsky.social) and César Córdova.

12/13 We plan to maintain and expand PertEval, creating a comprehensive benchmarking suite for the research community. Community contributions are very much encouraged!

Paper 📃: www.biorxiv.org/content/10.1...
GitHub 💻: github.com/aaronwtr/Per...

11/13 Looking ahead, we believe progress in this field will specifically require two key elements:
- Higher-quality data spanning a wider range of cellular states and perturbations
- Specialized models designed to fully leverage large-scale datasets for perturbation prediction

10/13 These findings highlight important challenges in using scFMs for perturbation effect prediction. While scFMs have potential, our results suggest that current models aren't yet optimized for this specific task.

9/13 Our analysis revealed that all models struggle to predict strong or atypically distributed perturbations and mostly learn average perturbation effects in a zero-shot setting. This highlights the need for training data that better represents cellular states and responses to perturbations.

8/13 Many perturbation prediction evaluations use 2,000 HVGs, while most genes don't show a strong response. However, even when narrowing down to the top 20 DEGs per perturbation, some scFM embeddings only slightly outperformed the baseline methods, while others still didn't.

7/13 We found that current-generation zero-shot scFM embeddings showed no significant improvement over task-specific model GEARS or even over simple baselines when predicting perturbation effects across 2,000 highly variable genes (HVGs).

6/13 On top of this, our framework also considers distribution shift, a frequently overlooked factor. We applied PertEval to evaluate zero-shot embeddings from several scFMs: scBERT, Geneformer, scGPT, scFoundation, and UCE.

5/13 PertEval-scFM includes three metrics:

- Area Under the SPECTRA Performance Curve (AUSPC)
- E-distance
- Pre-train / fine-tune cosine similarity (contextual alignment)

Each metric provides unique insights into model behaviour and robustness.

4/13 Our framework introduces a standardized toolkit of metrics designed to provide a nuanced evaluation of perturbation effect prediction model performance. Such a framework facilitates meaningful comparisons across different approaches and datasets.

3/13 Currently, there's no agreement on how to compare different approaches for perturbation effect prediction. This makes it challenging to determine which models truly perform best, or to identify areas for improvement. PertEval-scFM aims to change that.

2/13 With the rapid rise of models and scFMs for this task, it's more important than ever to have standardized evaluation methods. PertEval-scFM provides a comprehensive framework to assess these AI models in predicting cellular responses to genetic perturbations.

Thank you for your great work. We have a copy here at the office proudly on display 😎

Excited to attend this, looking forward to it!

My first time submitting to a big ML conference. Very frustrating experience after having worked really hard to address all the reviewers’ concerns only to be met with silence once we completed and shared the results. Hoping the meta-reviews will be better