Awesome, I am glad to hear this!

Happy reading and coding! Regarding the calculus part, I do have something here :)

sebastianraschka.com/pdf/books/dl...

currently halfway done with chapter 4 😁

I like trying new and different things. I don’t have as many convos here, but those I had were quite insightful, so I am hoping for more of that!

haha you never now on the internet these days. But joking aside, I did update the article with MiniMax M2 and Kimi Linear last week :)
magazine.sebastianraschka.com/p/the-big-ll...

Had that on my list for this wknd!

This is from a PyTorch developer’s perspective. For that, it’s great so far!

Kimi came 2 days later 😆

ha, very timely! Just got back from the conference and haven't had a chance to read the M2 report. But based on the Model Hub, it seems that SWA is not the default (similar to the recent Mistral Models) 🤔
(Source: huggingface.co/MiniMaxAI/Mi...)

there are trade-offs, but I find it refreshing that people are working on this :)

You could, but I would not recommend 😅
bsky.app/profile/seba...

Yes, if you set n_kv_groups = n_heads then it's multi-query attention. But it's not recommended as it too much of an extreme case and results in poor modeling performance. I don't know any LLM using it.

But keep in mind that they are special-purpose models, not text models. But that's also the whole selling point: small special purpose models instead of LLMs.

Interesting! I am not sure it will be a concatenation of models (I can't see how it would work), but I can see using modules like this that a generalist model can call for specific problems.