@andykeller.bsky.social @kempnerinstitute.bsky.social presented “Flow Equivariant Cybernetics”, a blueprint for agents that learn through continuous feedback with their environment.

Such a cool connection!! I never heard of that, but that is an ingenious solution. I will likely use this reference in my future talks and mention your comment if you don’t mind!

Thanks for reading! Can you explain your thought process here? Imagine a neuron with a receptive field (size of the yellow square) localized to the center of the pentagon. Its input would be entirely white — same as if it were localized to the center of the triangle; and therefore indistinguishable.

And not to forget, a huge thanks to all those involved in the work: Lyle Muller, Roberto Budzinski & Demba Ba!! And further thanks to those who advised me and shaped my thoughts on these ideas @wellingmax.bsky.social & Terry Sejnowski. This work would not have been possible without their guidance.

For all the technical details and more ablations, please see our paper recently accepted in workshop-form at ICLR Re-Align, and full-version preprint on ArXiv!

Paper: arxiv.org/abs/2502.06034
Code: github.com/KempnerInsti...

Hope to see you in Singapore!

Fin/

If you want more visualizations, a bit more depth, and even some audio of what different images 'sound' like to our models, please check out our @kempnerinstitute.bsky.social blog-post!

kempnerinstitute.harvard.edu/research/dee...

13/14

Overall, we believe this is the first step of many towards creating neural networks with alternative methods of information integration, beyond those that we have currently such as network depth, bottlenecks, or all-to-all connectivity, like in Transformer self-attention.

12/14

We found that wave-based models converged much more reliably than deep CNNs, and even outperformed U-Nets with similar numbers parameter when pushed to their limits. We hypothesize that this is due to the parallel processing ability that wave-dynamics confer and other CNNs lack.

11/14

As a first step towards the answer, we used the Tetris-like dataset and variants of MNIST to compare the semantic segmentation ability of these wave-based models (seen below) with two relevant baselines: Deep CNNs w/ large (full-image) receptive fields, and small U-Nets.

10/14

We were super excited about these results—they aligned with the long-standing hypothesis that traveling waves integrate spatial information in the brain*. But does this hold any practical implications for modern machine learning?

pubmed.ncbi.nlm.nih.gov/7947408
www.science.org/doi/abs/10.1...

9/14

Was this just due to using Fourier transforms for semantic readouts, or wave-biased architectures? No! The same models with LSTM dynamics and a linear readout of the hidden-state timeseries still learned waves when trying to semantically segment images of Tetris-like blocks!

8/14

Looking at the Fourier transform of the resulting neural oscillations at each point in the hidden state, we then saw that the model learned to produce different frequency spectra for each shape, meaning each neuron really was able to 'hear' which shape it was a part of!

7/14

We made wave dynamics flexible by adding learned damping and natural frequency encoders, allowing hidden state dynamics to adapt based on the input stimulus. On simple polygon images, we found the model learned to use these parameters to produce shape-specific wave dynamics:

6/14

To test this, we needed a task; so we opted for semantic segmentation on large images, but crucially with neurons having very small one-step receptive fields. Thus, if we were able to decode global shape information from each neuron, it must be coming from recurrent dynamics.

5/14

We found that, in-line with theory, we could reliably predict the area of the drum analytically by looking at the fundamental frequency of oscillations of each neuron in our hidden state. But is this too simple? How much further can we take it if we add learnable parameters?

4/14

Inspired by Mark Kac’s famous question, "Can one hear the shape of a drum?" we thought: Maybe a neural network can use wave dynamics to integrate spatial information and effectively "hear" visual shapes... To test this, we tried feeding images of squares to a wave-based RNN:

3/14