Also as O1 style reasoning datasets "comes online" in
the dataset space

We plan to do more training on these new line of QRWKV and LLaMA-RWKV models, over larger context lengths so that they can be true transformer killer

If your @ Neurips, you can find me with an RWKV7 Goose

This is in addition to our latest candidate RWKV-7 "Goose" 🪿 architecture. Which we are excited for, as it shows early signs of a step jump from our v6 finch 🐤 models.

Which we are scheduled to do a conversion run as well for 32B, and 70B class models

x.com/BlinkDL_AI/s...

Why is this important?

With the move to inference time thinking (O1-reasoning, chain-of-thought, etc). There is an increasing need for scalable inference over larger context lengths

The quadratic inference cost scaling of transformer models is ill suited for such long contexts

QKV Attention is **not** all you need

We release QRWKV6-32B-Instruct preview, a model converted from Qwen-32B instruct, trained for several hours on 2 MI300 nodes.

Surpassing all previous known open linear models (StateSpace, Hybrid, etc)

Unlocking 1000x+ lower inference cost

So yea, we just finished up the best subquadratic model for our QRWKV varient. Landing in hot during neurips

Matching transformer level performance despite the lack of "Quadratic Attention", using RWKV Attention instead

Proving Attention is **not** all you need

Finally done with work meetings and got to chill and slowly experience proper Canadian food:

Poutine and beer at a bar

Discord Quebec gang: Toronto Poutine ain’t real Poutine 🤣

(Will be back in SF tomorrow)

Just landed in Toronto 🇨🇦

Me as South East Asian: oooo… snow ☃️
My Canadian friends: that’s barely any snow ❄️

The true Canadian experience needs to have at least knee high snow I guess 🤣

So i heard ChatGPT is looking into adding ads now ...
( the circle of internet life )

I have only one goal this #OpenAI #devday

To help build up the Singapore AI community
And somethings are looking promising

(🇸🇬/acc)