We then apply this approach to 3 case studies - prediction by part-of-speech, multi-token fact recall, and fixed-response question answering. Check the paper & Akshat's thread for details!

(look at this model predicting positive/negative sentiment - such a clear pattern!)

4/

Most early layer predictions get overturned! It appears that these are statistical guesses, made when the model has not yet processed enough contextual information.

(flip rate = how often the model's final prediction differs from the current layer's prediction)

3/

In early layers, most frequent tokens (e.g., "the") dominate predictions, whereas infrequent tokens gradually become more predicted later on.

2/

🤖📈 How do LLMs use their depth?

Akshat Gupta led a fun project to find out! We leverage TunedLens (~linear decoding of tokens) to explore how LLMs' internal representations change from layer to layer.

Preprint: arxiv.org/abs/2510.18871

1/

Finally out in TACL:
🌎EWoK (Elements of World Knowledge)🌎: A cognition-inspired framework for evaluating basic world knowledge in language models

tl;dr: LLMs learn basic social concepts way easier than physical&spatial concepts

Paper: direct.mit.edu/tacl/article...
Website: ewok-core.github.io

Many thanks to the volunteer organizers and the flash talk presenters for making the CCN watch party at Georgia Tech a success! And thanks all attendees for coming and engaging in discussions!

Looking forward to #CogSci2025 ! Find us throughout the conference

Atlanta community: we are organizing a CCN @cogcompneuro.bsky.social watch party!

Watch the talks, engage in structured discussions, and (optionally) present your own work.

Register:
forms.gle/AWxVPbrgxkdd...
Schedule:
tinyurl.com/ccn2025atlanta

My commentary on the do's and don'ts of cognitive evaluations in LLMs is now out in Nature Human Behavior:

doi.org/10.1038/s415...

posting here with a figure that didn't make it into the final draft and is now instead a boring table :P

#CogSci #LLMs #AI