11/13 …and also outperform a wealth of state-of-the-art neuro-AI ANNs in predicting visually-evoked brain activity, despite being trained from scratch on a dataset that is orders of magnitude smaller.

10/13 Finally, training Artificial Neural Networks (ANNs) to predict LLM embeddings of scene captions provides highly competitive models of the visual system. LLM-trained ANNs better match brain activities than stringently controlled category-trained models…

9/13 Contrasting models that differ in their ability to encode contextual information in scene captions shows that the match of LLMs with visual brain activities is linked to their ability to integrate and contextualise information beyond category words or the caption nouns.

7/13 We can also train a linear decoding model to predict LLM embeddings from fMRI voxels. This allows us to decode remarkably accurate textual descriptions of the stimuli viewed by participants based on their brain activity alone, all with a simple linear model!

6/13 Using our encoding model, we can write any sentence and predict visual brain activities that would be evoked if the participant saw an image captioned by that sentence. We can localise face/place/food areas just by writing two sentences and contrasting predicted activities!

5/13 We use RSA and encoding models to quantify the match of LLM embeddings of scene captions with brain activities on the NSD dataset. We find a strong match with visually evoked brain responses, especially in higher level areas of the ventral, lateral and parietal streams.

How is this arrangement instantiated? Via the feedforward sweep. If we train feedforward NNs, across images, the norms in their pre-readout layers are negatively correlated with the t_stable values in the RNN i.e. images classified earlier start with higher activation norms. 8/10

This is what we found! The representations of images with t_stable <= t are further away from the decision boundaries via two mechanisms: their norm is higher and they are closer to their readout vectors.

This holds true across recurrent connection and interaction types. 7/10

Given the equal slowing down, how does the RNN manage to change the class of incorrectly classified images while maintaining the correctness of the other images? 

Proposal: at a given t, the images with t_stable <= t should be further away from the decision boundaries. 6/10

Surprisingly, no! The change in representations decreases but there's no relationship with t_stable. The changes all slow down at the same rate.

Unrolling the RNNs beyond the timesteps of training revealed that these RNNs indeed have convergent dynamics. 5/10

We trained RNNs with convolutional layers (termed BLT 🍔 - github.com/KietzmannLab...) to classify images from MiniEcoset (osf.io/msna2/) into 100 classes and analyzed their internal dynamics. 3/10