It's still an awesome thing to use as it allows you to create hypotheses on what could be wrong with your model and making models better is in my opinion about making hypotheses, and testing them. This adds one more tool for it.

Let's take metric from the blog information abundance.
1. Everything is relative.. if your model is trash you can't trust it
2. Getting a number like random means you should increase your embedding
3. A number below one means reducing
It compares to random but details matters (uniform, normal...?)

Thanks for the paper. We were wondering this at our job and it reinforces our gut feeling.
I agree it would be interesting comparing different kind of models (late interaction, sparse, hybrid like SparseEmbed...).
Will you consider sharing the code?

The paper also links to arxiv.org/abs/2308.08487. Beside TIN which allows you to do attention for your historical data. To me a big takeaway is CTC metric which allows you to measure the temporal correlation of your model. Another tool you can use to debug your model.

One of the topics of this paper is embedding collapsing. Since then it has been covered by more blogs (eg. blog.reachsumit.com/posts/2024/1...). I find it very interesting personally. Debugging models is hard and this gives you a way to potentially find issues with a simple tensorboard callback.

Aerospace (github.com/nikitabobko/...) has been a productivity boost for me.