Chris Olah
@colah.bsky.social
Reverse engineering neural networks at Anthropic. Previously Distill, OpenAI, Google Brain.Personal account.
The interpretability team will be mentoring more fellows this cycle, so if you're interested in interpretability, it might be worth applying!
Some of our fellows last cycle did this: arxiv.org/pdf/2507.21509
Some of our fellows last cycle did this: arxiv.org/pdf/2507.21509
arxiv.org
August 12, 2025 at 6:54 PM
The interpretability team will be mentoring more fellows this cycle, so if you're interested in interpretability, it might be worth applying!
Some of our fellows last cycle did this: arxiv.org/pdf/2507.21509
Some of our fellows last cycle did this: arxiv.org/pdf/2507.21509
But more importantly, I hope it will just help clarify what we mean by interference weights!
July 29, 2025 at 11:33 PM
But more importantly, I hope it will just help clarify what we mean by interference weights!
Our new note demonstrates that interference weights in toy models can demonstrate strikingly similar phenomenology to that of Towards Monosemanticity...
July 29, 2025 at 11:33 PM
Our new note demonstrates that interference weights in toy models can demonstrate strikingly similar phenomenology to that of Towards Monosemanticity...
They've been an ongoing challenge in our work for a long time. In fact, our recent work on attribution graphs (transformer-circuits.pub/2025/attribu...) was partly designed as a method to side step them as a challenge!
Circuit Tracing: Revealing Computational Graphs in Language Models
We describe an approach to tracing the “step-by-step” computation involved when a model responds to a single prompt.
transformer-circuits.pub
July 29, 2025 at 11:33 PM
They've been an ongoing challenge in our work for a long time. In fact, our recent work on attribution graphs (transformer-circuits.pub/2025/attribu...) was partly designed as a method to side step them as a challenge!
The keen reader may recall all these plots referencing "interference weights??" in Towards Monosemanticity (transformer-circuits.pub/2023/monosem...).
July 29, 2025 at 11:33 PM
The keen reader may recall all these plots referencing "interference weights??" in Towards Monosemanticity (transformer-circuits.pub/2023/monosem...).
I should also mention that I wrote a blog post listing a bunch of specific analogies between deep learning and biology several years back. (It's probably of much narrower interest!)
colah.github.io/notes/bio-an...
colah.github.io/notes/bio-an...
Analogies between Biology and Deep Learning [rough note]
A list of advantages that make understanding artificial nerural networks much easier than biological ones.
colah.github.io
May 13, 2025 at 7:34 PM
I should also mention that I wrote a blog post listing a bunch of specific analogies between deep learning and biology several years back. (It's probably of much narrower interest!)
colah.github.io/notes/bio-an...
colah.github.io/notes/bio-an...
Of course, I'd be remiss to not mention that many others have made analogies between work in machine learning and biology -- most notable for us is the "bertology" work, which framed it self as studying the biology of the BERT models.
May 13, 2025 at 7:34 PM
Of course, I'd be remiss to not mention that many others have made analogies between work in machine learning and biology -- most notable for us is the "bertology" work, which framed it self as studying the biology of the BERT models.
But we also think it's important for such "biology" results (which are more foreign in style to machine learning) to be treated as worthy of publication independent of methods work (which looks more similar to normal machine learning).
May 13, 2025 at 7:34 PM
But we also think it's important for such "biology" results (which are more foreign in style to machine learning) to be treated as worthy of publication independent of methods work (which looks more similar to normal machine learning).
This was partly a convenient way to handle the length (jointly, the two papers are ~150 pages!).
May 13, 2025 at 7:34 PM
This was partly a convenient way to handle the length (jointly, the two papers are ~150 pages!).
But why did the language come up in our paper title? There was actually a further reason, which is that we wanted to separate our "methods" work and what we called our "biology" work (i.e. the empirical research we did using our method).
May 13, 2025 at 7:34 PM
But why did the language come up in our paper title? There was actually a further reason, which is that we wanted to separate our "methods" work and what we called our "biology" work (i.e. the empirical research we did using our method).
Finally, you need to believe that a worthy mode of investigation is empirical (rather than theoretical), and a style of empirical research that's more open to the qualitative than purely quantitative.
This evokes biology more than physics.
This evokes biology more than physics.
May 13, 2025 at 7:34 PM
Finally, you need to believe that a worthy mode of investigation is empirical (rather than theoretical), and a style of empirical research that's more open to the qualitative than purely quantitative.
This evokes biology more than physics.
This evokes biology more than physics.
One further needs to believe that individual neural networks, and in fact sub-components of those networks, warrant investigation. That's more idiosyncratic!
May 13, 2025 at 7:34 PM
One further needs to believe that individual neural networks, and in fact sub-components of those networks, warrant investigation. That's more idiosyncratic!
At a basic level, one needs to believe deep learning warrants scientific investigation. This doesn't seem very controversial these days, but note that it's already kind of radical. See eg. Herbert Simon's The Sciences of the Artificial.
May 13, 2025 at 7:34 PM
At a basic level, one needs to believe deep learning warrants scientific investigation. This doesn't seem very controversial these days, but note that it's already kind of radical. See eg. Herbert Simon's The Sciences of the Artificial.
I've written multiple papers characterizing (small sets of) individual neurons. Historically, this hasn't seemed like a worthy topic of a paper in ML – I've had to justify it!
May 13, 2025 at 7:34 PM
I've written multiple papers characterizing (small sets of) individual neurons. Historically, this hasn't seemed like a worthy topic of a paper in ML – I've had to justify it!
One way in which this is important is that the *types of questions* we're interested in are quite bizarre from a traditional machine learning perspective, but natural under the biological frame.
May 13, 2025 at 7:34 PM
One way in which this is important is that the *types of questions* we're interested in are quite bizarre from a traditional machine learning perspective, but natural under the biological frame.
I think there's a deep way in which the scientific aesthetic of biology is very relevant to deep learning and especially interpretability.
Biology is to evolution as interpretability is to gradient descent.
bsky.app/profile/cola...
Biology is to evolution as interpretability is to gradient descent.
bsky.app/profile/cola...
The elegance of ML is the elegance of biology, not the elegance of math or physics.
Simple gradient descent creates mind-boggling structure and behavior, just as evolution creates the awe inspiring complexity of nature.
x.com/banburismus_...
Simple gradient descent creates mind-boggling structure and behavior, just as evolution creates the awe inspiring complexity of nature.
x.com/banburismus_...
x.com
May 13, 2025 at 7:34 PM
I think there's a deep way in which the scientific aesthetic of biology is very relevant to deep learning and especially interpretability.
Biology is to evolution as interpretability is to gradient descent.
bsky.app/profile/cola...
Biology is to evolution as interpretability is to gradient descent.
bsky.app/profile/cola...
Stepping back, "physics of neural networks" is a whole area of research. Of course, it isn't physics in a classical sense. It's bringing the methods and style of physics to deep learning.
We refer to the "biology" of neural networks in a similar spirit!
We refer to the "biology" of neural networks in a similar spirit!
May 13, 2025 at 7:34 PM
Stepping back, "physics of neural networks" is a whole area of research. Of course, it isn't physics in a classical sense. It's bringing the methods and style of physics to deep learning.
We refer to the "biology" of neural networks in a similar spirit!
We refer to the "biology" of neural networks in a similar spirit!
My colleagues and I have actually been using "biology" quite heavily as a metaphor and handle for several years now, beyond the title of this paper. There are a lot of reasons I think it's useful!
May 13, 2025 at 7:34 PM
My colleagues and I have actually been using "biology" quite heavily as a metaphor and handle for several years now, beyond the title of this paper. There are a lot of reasons I think it's useful!
(This is a cross-post of one of my favorite old twitter threads: x.com/ch402/status... )
Chris Olah on X: "The elegance of ML is the elegance of biology, not the elegance of math or physics. Simple gradient descent creates mind-boggling structure and behavior, just as evolution creates the awe inspiring complexity of nature." / X
The elegance of ML is the elegance of biology, not the elegance of math or physics. Simple gradient descent creates mind-boggling structure and behavior, just as evolution creates the awe inspiring complexity of nature.
x.com
May 13, 2025 at 7:32 PM
(This is a cross-post of one of my favorite old twitter threads: x.com/ch402/status... )
Every model is its own entire world of beautiful structure waiting to be discovered, if only we care to look.
May 13, 2025 at 7:32 PM
Every model is its own entire world of beautiful structure waiting to be discovered, if only we care to look.
I wish people would spend more time looking at the models we create though. It's like we're launching expeditions with complex equipment to reach more and more remote islands and tall mountains... and the biology stops at measuring the size and weight of the animals we find.
May 13, 2025 at 7:32 PM
I wish people would spend more time looking at the models we create though. It's like we're launching expeditions with complex equipment to reach more and more remote islands and tall mountains... and the biology stops at measuring the size and weight of the animals we find.