In summary: We analytically study the dynamics of localization in a nonlinear neural network without top-down constraints, where we find that “edges” drive localization. We identified which forms of non-Gaussianity were necessary to get this structure to emerge. (9/10)

If we depart from a property of our idealized images, we can’t maintain this clear analytical picture. However, we’re still able to show localization never emerges for elliptically-distributed data. This is a broad class of distributions that can be highly non-Gaussian! (8/10)

Looking closer, we identify a key statistic driving localization: excess kurtosis of the pixel marginals. When it’s negative (i.e., the data have sharper “edges”), localization emerges; otherwise, localization is suppressed. This holds beyond our single ReLU neuron, too. (7/10)

The localization amplifier depends only on a particular attribute of the data—the pixel marginal distributions. When they’re Gaussian, the amplifier is a linear function. But, when they’re concentrated, it becomes super-linear. This promotes localization! (6/10)

Doing theory here is tricky, as IG22 discuss: nonlinear, non-Gaussian feature learning is complicated! We derive the learning dynamics of a minimal case—a single ReLU neuron learning from an idealized model of images—isolating a data-driven “localization amplifier.” (5/10)