8/n 🔬 Interesting result: proteins with high likelihoods from MSA-based estimates tend to have more deleterious mutations — a pattern not observed for ESM2.

7/n 🔬 Interesting result: random seeds also impact predicted sequence likelihoods.

6/n 🔬 Interesting result: when scale up model size, predicted sequence likelihoods vary across proteins — some show no improvement, while others show substantial gains.⚡

5/n 🎯 pLMs align best with MSA-based per-residue likelihoods at moderate levels of sequence likelihood, which explains the bell-shaped relationship we observed in fitness prediction performance.

4/n 🔍 Interestingly, while overall sequence likelihoods differ, per-residue predicted likelihoods are correlated.

The better a pLM’s per-residue likelihoods align with MSA-based estimates, the better its performance on fitness prediction. 📈

3/n 🧬 Predicted likelihoods from MSA-based methods directly reflect evolutionary constraints.

Even though pLMs are also trained to learn evolutionary information, their predicted whole sequence likelihoods show no correlation with MSA-based methods. ⚡

2/n 📊 We found that general pLMs trained on large datasets show a bell-shaped relationship between fitness prediction performance and wild-type sequence likelihood.

Interestingly, MSA-based models do not show this trend.

1/n 🧬 These models estimate sequence likelihood from different inputs — sequence, structure, and homologs.

To infer mutation effects, the log-likelihood ratio (LLR) between the mutated and wild-type sequences is used. ⚖️

We just updated our manuscript "Understanding Language Model Scaling on Protein Fitness Prediction". Where we explained why larger pLMs don’t always perform better on mutation effect prediction. We extended beyond ESM2 to models like ESMC, ESM3, SaProt, and ESM-IF1.

#ProteinLM

⚠️ Caution when using ProteinGYM binary classification: some DMS_binarization_cutoff values appear inverted, some are totally unreasonable given the DMS_score shows a clear bimodal distribution.

Why do large protein language models like ESM2-15B underperform compared to medium-sized ones like ESM2-650M in predicting mutation effects? 🤔

We dive into this issue in our new preprint—bringing insights into model scaling on mutation effect prediction. 🧬📉

4/n We also compared SeqDance's attention with ESM2-35M.

3/n here is the comparison on viral proteins in ProteinGYM, (our models have 35M parameters)

2/n on the mega-scale protein stability dataset, it's clear that ESM2's performance is correlated with the number of homologs in its training set. but our models show robust performance for proteins without homologs in training set.

1/n to perform zero-shot fitness prediction, we use our models SeqDance/ESMDance to predict dynamic properties of both wild-type and mutated sequences. the relative changes bettween them are used to infer mutation effects.

We have updated our protein lanuage model trained on structure dynamics. Our new models show significant better zero-shot performance on mutation effects of designed and viral proteins compared to ESM2. check the new preprint here: www.biorxiv.org/content/10.1...