I did my first classical sangati this weekend! Came a long way from banging on tables as a kid :)

We thus see the emergence of two regimes, one where we have a lower no. of hypotheses (where the Bayesian approach is unmatched) and another with a higher no. of hypotheses (where transformers take the lead).

While the transformer is heavier for lower lookback, the compute of the Bayesian method increases super-exponentially on increasing lookback! This is a perfect illustration of our combinatorial challenge of tracking and how transformers could help in resolving it.

Not only is the transformer suboptimal, it remains suboptimal when the Bayesian method is optimal (hint: AI alignment problem). Increasing the amount of data starts decreasing the accuracy!

But what if we had a world where this was possible (i.e., short sequences of 8 time steps, hence less no. of hypotheses)? No matter how much we train the transformer, it never matches the optimal performance!

Transformers are robust when dealing with large information. On increasing noise (for 2 particles undergoing brownian motion for 150 timesteps) we see a prolongation in the breakpoint of accuracy in all cases. An increase in sequence lookback shows further prolongation!