[7/n] Our architecture allows us to know what concepts the model learned and which concepts the model uses during inference by inspecting the weights of the final linear layer; this offers a way to debug and asses the model's quality.

[6/n] Interpretability: The concept bottleneck can be used to understand which concept the model uses to predict a certain amino acid. Reliably controlling model behavior: The concepts can be used as knobs to control the model's output.

[5/n] We train a mask language model with up to 3 Billion parameters with a layer that directly encodes biophysical and biochemical concepts that biologists care about. These models match the performance of unconstrained masked language model.

[3/n] In our concept bottleneck protein language model paper, we show that we can train the model with billions of parameters, with interpretability constraints, without performance degradation.