Bonnie Berger Lab
@bergerlab.bsky.social
The Berger lab at @csail.mit.edu works on a diverse set of problems in computational biology and biomedicine. Account run by lab members. https://people.csail.mit.edu/bab/
7/ We show that SAE & transcoder features are much more interpretable than ESM neurons, for both protein-level & amino acid-level representations. This has the potential to improve safety, trust & explainability of PLMs. As PLMs improve, SAEs could help us learn new biology.
September 3, 2025 at 4:51 PM
7/ We show that SAE & transcoder features are much more interpretable than ESM neurons, for both protein-level & amino acid-level representations. This has the potential to improve safety, trust & explainability of PLMs. As PLMs improve, SAEs could help us learn new biology.
6/ We also use Claude to autointerpret SAE features based on protein names, families, gene names & GO terms. Many features correspond to families (like NAD Kinase, IUNH, PTH) & functions (like methyltransferase activity, olfactory/gustatory perception).
September 3, 2025 at 4:51 PM
6/ We also use Claude to autointerpret SAE features based on protein names, families, gene names & GO terms. Many features correspond to families (like NAD Kinase, IUNH, PTH) & functions (like methyltransferase activity, olfactory/gustatory perception).
5/ We interpret these SAE features using Gene Ontology (GO) enrichment. Many protein-level SAE features align tightly with GO terms across all levels of the GO hierarchy.
September 3, 2025 at 4:51 PM
5/ We interpret these SAE features using Gene Ontology (GO) enrichment. Many protein-level SAE features align tightly with GO terms across all levels of the GO hierarchy.
4/ SAEs have a very wide latent dimension with a sparsity constraint. This forces PLM representations to disentangle into biologically interpretable, sparsely activating features without any supervision.
September 3, 2025 at 4:50 PM
4/ SAEs have a very wide latent dimension with a sparsity constraint. This forces PLM representations to disentangle into biologically interpretable, sparsely activating features without any supervision.
3/ We train sparse autoencoders (SAEs) on protein-level and amino acid-level representations from layers 6-10 of ESM2_t12_35M_UR50D. We also train transcoders (an SAE variant) on protein-level representations.
September 3, 2025 at 4:50 PM
3/ We train sparse autoencoders (SAEs) on protein-level and amino acid-level representations from layers 6-10 of ESM2_t12_35M_UR50D. We also train transcoders (an SAE variant) on protein-level representations.
2/ Protein-level representations from PLMs are used in many downstream tasks. Disentangling their features can enhance interpretability, helping us trust and explain downstream applications.
September 3, 2025 at 4:50 PM
2/ Protein-level representations from PLMs are used in many downstream tasks. Disentangling their features can enhance interpretability, helping us trust and explain downstream applications.
1/ PLMs like ESM have made big strides in predicting protein structure & function. But they feel like a “black-box.” What biological information do PLM representations contain? Can we disentangle them systematically?
September 3, 2025 at 4:50 PM
1/ PLMs like ESM have made big strides in predicting protein structure & function. But they feel like a “black-box.” What biological information do PLM representations contain? Can we disentangle them systematically?