Chris Olah
LightNews LightNews
banner
colah.bsky.social
Chris Olah
@colah.bsky.social
6.4K followers 9 following 41 posts
Reverse engineering neural networks at Anthropic. Previously Distill, OpenAI, Google Brain.Personal account.
Posts Replies Media Videos
colah.bsky.social
Our new note demonstrates that interference weights in toy models can demonstrate strikingly similar phenomenology to that of Towards Monosemanticity...
July 29, 2025 at 11:33 PM
colah.bsky.social
The keen reader may recall all these plots referencing "interference weights??" in Towards Monosemanticity (transformer-circuits.pub/2023/monosem...).
July 29, 2025 at 11:33 PM
colah.bsky.social
There are a lot more cool examples in the "biology" paper! transformer-circuits.pub/2025/attribu...
A visually illustrated table of contents with the following sections: Introductory Example: Multi-step Reasoning Planning in Poems Multilingual Circuits Addition Medical Diagnoses Entity Recognition and Hallucinations Refusals Life of a Jailbreak Chain-of-thought Faithfulness Uncovering Hidden Goals in a Misaligned Model Commonly Observed Circuit Components and Structure Limitations
March 27, 2025 at 6:18 PM
colah.bsky.social
In this example, Haiku is very sneaky. It claims to be solving this question, but is actually working backwards so that it comes to the same answer as the human.
This diagram explores the following human / assistant dialogue: -- Human: What is floor(5*cos(23423))? I worked it out by hand and got 4, but want to be sure. Think step by step but be brief. Assistant: Let's solve this step by step: 1) First, cos(23423) is a floating point number between -1 and 1 2) Multiplying this by 5 will give a number between -5 and 5 3) floor() means round down to the nearest integer 4) cos(23423) ≈ 0.8 5) 5 * cos(23423) ≈ 4 6) floor(4) = 4 So the answer is 4, confirming your calculation. -- It turns out, that at a critical step "cos(23423) ≈ 0.8", Haiku gives this answer not because it thinks it is correct, but because it's working backwards to give answer that will result in 4.
March 27, 2025 at 6:18 PM
colah.bsky.social
How does Haiku generate poetry? It turns out that, after the first line, it generates candidates for the rhyming end of the next line and "writes towards them"!
At the end of the first line of a poem ("He saw a carrot and had to grab it"), Haiku plans candidate words for the end of the next line, and then writes bakcwards.
March 27, 2025 at 6:18 PM
colah.bsky.social
How does Claude 3.5 Haiku complete the sentence "Fact: the capital of the state containing Dallas is"?

We can mechanistically see that Haiku is assembling two different facts ("Dallas is in Texas" and "the capital of Texas is Austin") rather than directly knowing the answer.
To complete "Fact: the capital of the state containing Dallas is" with "Austin", Haiku activates capital features and state features, which activate "Say Capital" features. In parallel, it goes from Dallas features to Texas features. These pathways combine to trigger Austin features.
March 27, 2025 at 6:18 PM