We tried the first thing you mentioned few years ago arxiv.org/abs/2205.14794
Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning
Recurrent neural networks have a strong inductive bias towards learning temporally compressed representations, as the entire history of a sequence is represented by a single vector. By contrast, Trans...
arxiv.org
November 27, 2024 at 2:07 PM
We tried the first thing you mentioned few years ago arxiv.org/abs/2205.14794