We also looked at attention head outputs vs the residual stream, and find that linguistic information lives primarily in the residual stream.

Read the paper for our full analysis including steering vectors, intrinsic dimensionality, and training dynamics (in OLMo2 and Pythia)!

The probes show that lexical info concentrates in early layers and becomes increasingly nonlinear deeper in the network. Inflectional info remains linearly accessible throughout all layers.

The hierarchy is remarkably stable across all models:

Early layers capture syntax (POS, dependencies) -> Middle layers handle semantics & entities (NER, SRL) -> Later layers encode discourse (coreference, relations)

This holds whether you're looking at BERT, Qwen2.5 or OLMo 2.

🔬We probed 25 models, across 8 linguistic tasks spanning syntax, semantics, and discourse.

🔍The answer: YES. Modern LMs consistently rediscover the classical NLP pipeline.

🚨 New paper w/ @nsubramani23.bsky.social !

In 2019, Tenney et al. found that BERT rediscovers the classical NLP pipeline, offering linguistic insights into early LMs.

But models today are >>100x larger, trained on vastly more data…

Do they still rediscover the classical NLP pipeline? 🧵

We also tested how tokenization affects linguistic representations using analogy tasks (king - man + woman = ?) 👑

Whole-word embeddings consistently outperform averaged subtoken representations - linguistic regularities are stored at the word level, not compositionally!

🔬 We also measured intrinsic dimensionality across layers using PCA.

🎢 Some models (GPT-2, OLMo-2) compress their middle layers to just 1-2 dimensions capturing 50-99% of variance, then expand again! This bottleneck aligns with where grammar is most accessible & lexical info is most nonlinear.

To understand when these patterns emerge, we analyze OLMo-2 & Pythia checkpoints throughout pre-training. 👶👦👨👨‍🦳

We find that models learn this linguistic organization in the first few thousand steps! But this encoding slowly degrades as training progresses. 📉

🤔 But are classifiers actually learning linguistic patterns or just memorizing?

📈 We ran control tasks with random labels - inflection classifiers show high selectivity (real learning!) while lemma classifiers don't (memorization).

Key findings 📊:
- 📉 Lexical info concentrates in early layers & becomes increasingly nonlinear in deeper layers
- ✨ Inflection (grammar) stays linearly accessible throughout ALL layers
- Models memorize word identity but learn generalizable patterns for inflections!

🧐 How do modern LMs encode linguistic information? Do they represent words grouped by meaning (walk/walked) or grammar (walked/jumped)?

We trained classifiers on hidden activations from 16 models (BERT -> Llama 3.1) to find out how they store word identity (lexemes) vs. grammar (inflections).

🚨New #interpretability paper with @nsubramani23.bsky.social: 🕵️ Model Internal Sleuthing: Finding Lexical Identity and Inflectional Morphology in Modern Language Models