[The 3D plot of ReLU above is from addxorrol.blogspot.com/2024/07/some... ]. Guido Montufar has an excellent paper showing that the number of regions in the piecewise-affine function defined this way is exponential in the depth of the network: proceedings.neurips.cc/paper_files/... 3/3.

I will use that for the deep learning class I teach. Feel free to reuse ! In general I love the "origami effect" explanation of why deep learning works well. This is particularly clear for "ReLU" non linearities. You can then view each layer as folding your space in two. 2/n

Hi Jonathan! 🙂