openreview submission: openreview.net/pdf?id=OJsMG...

[5/5] The attention maps of the fMRI encoder, extracted after multimodal alignment, correlate well with topographic maps of functional networks activated during movie-watching. Code, pre-trained models, and a pre-print will be released very soon! ☺️

[4/5] Critically, we show that SIM generalises effectively for fMRI decoding and image reconstruction across new subjects (unseen during training), new audio-visual stimuli (new movie scenes), and even new stimuli in new subjects!

[3/5] In SIM, we demonstrate that surface-based models can serve as backbones for encoding/decoding models of brain activity in movie-watching experiments by modelling the dynamics of functional connectivity and aligning fMRI, video, and audio through multimodal alignment.

[2/5] This work has been the latest research project of my PhD, by far the most interesting and a fantastic collaboration between great AI and Neuroimaging researchers @gabriben.bsky.social, @metricsemma.bsky.social @danielrueckert.bsky.social @lzjwilliams.bsky.social , Rob Leech, and Yourong Guo!