We apply our masked VAE approach to drosophila 🪰walking behavior, a macaque 🐒reach task & a synthetic dataset with access to the ground truth.

We propose calibration checks to evaluate the models’ uncertainty estimates, to avoid making confidently wrong predictions.

Check out the paper for more 🙂

We use masked VAEs to address two model desiderata in one model:

1. jointly modeling conditional distributions that are commonly targeted in neuroscience (e.g., encoding 🐭➡️🧠and decoding 🧠 ➡️🐭) and

2. accounting for low-dimensional dynamics underlying both neural activity and behavior. 🌀

7) Classifying sequences of neural population activity revealed that SCs and SCim neurons differentiate self-generated vs. object-generated looming stimuli beyond what is already accounted for by other covariates, such as speed. 🏃🏼‍ ⏰ 🗺️

6) Finally, to assess how mice distinguish self-generated vs. object-generated stimuli, we replay image sequences generated by the animal's own movement from previous VR trials, thus decoupling the visual motion from their current behavior.

5) Surprisingly, we found behavioural responses triggered by the VR environments emerging from the first day of exposure, with animals spontaneously slowing down as they got closer to the object. 🏃🏃🛑

4) We then looked at SC responses to looming stimuli generated when mice navigate the VR environment. We found vision-dominated responses in SCs while animal speed greatly influenced SCim responses.

3) First, we characterized responses to approaching objects at different speeds, targeting Superficial (SCs) and Intermediate (SCim) layers of the Superior Colliculus.

2) In a fun collaboration with fantastic Stefano Zucca, @ppjgoncalves.bsky.social, @jakhmack.bsky.social, @amansaleem.bsky.social, and Sam Solomon we assessed how neurons in Superficial (SCs) and Intermediate (SCim) layers represent looming stimuli by using an immersive Virtual Reality environment.

1) Some exciting science in turbulent times:

How do mice distinguish self-generated vs. object-generated looming stimuli? Our new study combines VR and neural recordings from superior colliculus (SC) 🧠🐭 to explore this question.

Check out our preprint doi.org/10.1101/2024... 🧵

11) LDNS is particularly promising for heterogeneous datasets without trial structure, which pose challenges for many LVMs.

LDNS successfully mimicked cortical data during attempted speech—a challenging task due to varying trial lengths.

10) Colorful latents are just so nice to look at, so we were glad to see that the LDNS latent space preserves behavioral information.

Both latents and PCs thereof reflect the reach direction of reaches used for conditioning.

9) LDNS allows for flexible conditioning on behavioral variables.

Diffusion models conditioned on either reach direction or velocity trajectories produce neural activity samples that are consistent with the queried behavior.

8) To increase the realism of generated spikes even further, we demonstrate how to equip LDNS with more expressive autoregressive observation models.

(this can be applied to any LVM trained with Poisson log-likelihood!)

7) We then moved to a classic monkey reach task and show that LDNS samples are indistinguishable to the human eye from real cortical data and accurately capture population level and single neuron statistics.

6) We validate that LDNS does what it’s supposed to on simulated spiking data.

LDNS perfectly captured firing rates & underlying dynamics and can length-generalize—producing faithful samples of 16 times the original training length.

5) But how does LDNS work?

The AE first maps spikes to time-aligned latents, which allows training flexible (un)conditional diffusion models on smoothly varying latents, circumventing the issue of diffusion models acting on discrete values.

4) Latent Diffusion for Neural Spiking data to the rescue ⛑️

LDNS combines 1) a regularized S4-based autoencoder (AE) with 2) diffusion in latent space, and can model diverse neural spiking data.

Here we consider 3 very different tasks:

1) With our @neuripsconf.bsky.social poster happening tomorrow, it's about time to introduce our Spotlight paper 🔦, co-lead with @jkapoor.bsky.social:

Latent Diffusion for Neural Spiking data (LDNS), a latent variable model (LVM) which addresses 3 goals simultaneously: