Ya maybe. I still think it's cool. E.g. you see experts specialized for groups of imagenet categories in vision MoEs. arxiv.org/abs/2106.05974

I feel like mainstream deep learning is actually coming around to specialization. Look at MoEs. The benefit is increased model capacity without increased compute.

I agree, it makes sense that you could get a kind of soft weight sharing through the development process. But still, when we count the weights of a CNN we just count the kernels. But in V1 we count every receptive field. I just feel like the brain and ANN counts are not exactly comparable.

If you consider that GPT4 copies its 10^12 parameters across its 10^5 tokens, then it already has 10^17 synapses, much larger than the brain. It's just the brain can't share weights like GPT, so every parameter is unique.