PicoCreator - AI Model Builder 🛫 NeurIPS
@picocreator.bsky.social
Serverless 🦙 @ https://featherless.ai
Build Attention-Killers AI (RWKV) from scratch @ http://wiki.rwkv.com
Also built uilicious & GPU.js (http://gpu.rocks)
Build Attention-Killers AI (RWKV) from scratch @ http://wiki.rwkv.com
Also built uilicious & GPU.js (http://gpu.rocks)
Reposted by PicoCreator - AI Model Builder 🛫 NeurIPS
happy year of the THIS GUY!! 🐍
January 2, 2025 at 3:32 AM
happy year of the THIS GUY!! 🐍
Reposted by PicoCreator - AI Model Builder 🛫 NeurIPS
I'll get straight to the point.
We trained 2 new models. Like BERT, but modern. ModernBERT.
Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc. Real practical stuff.
It's much faster, more accurate, longer context, and more useful. 🧵
We trained 2 new models. Like BERT, but modern. ModernBERT.
Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc. Real practical stuff.
It's much faster, more accurate, longer context, and more useful. 🧵
December 19, 2024 at 4:45 PM
I'll get straight to the point.
We trained 2 new models. Like BERT, but modern. ModernBERT.
Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc. Real practical stuff.
It's much faster, more accurate, longer context, and more useful. 🧵
We trained 2 new models. Like BERT, but modern. ModernBERT.
Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc. Real practical stuff.
It's much faster, more accurate, longer context, and more useful. 🧵
Random opinion on modern love, we need to normalize
- breaking up & being friend
- dating friend
- friend groups being chill & supportive, with all of it (getting together, or breaking up)
Starting romantic relations, without knowing your partner as a person is weird to me
bias: I married a friend
- breaking up & being friend
- dating friend
- friend groups being chill & supportive, with all of it (getting together, or breaking up)
Starting romantic relations, without knowing your partner as a person is weird to me
bias: I married a friend
December 16, 2024 at 7:40 PM
Random opinion on modern love, we need to normalize
- breaking up & being friend
- dating friend
- friend groups being chill & supportive, with all of it (getting together, or breaking up)
Starting romantic relations, without knowing your partner as a person is weird to me
bias: I married a friend
- breaking up & being friend
- dating friend
- friend groups being chill & supportive, with all of it (getting together, or breaking up)
Starting romantic relations, without knowing your partner as a person is weird to me
bias: I married a friend
My personal high belief / high conviction hot take:
32-72B is all we need for human level of AGI 🤖
Anything higher is just us being inefficient in architecture / code / etc
#NeurIPS2024
32-72B is all we need for human level of AGI 🤖
Anything higher is just us being inefficient in architecture / code / etc
#NeurIPS2024
December 12, 2024 at 10:44 PM
My personal high belief / high conviction hot take:
32-72B is all we need for human level of AGI 🤖
Anything higher is just us being inefficient in architecture / code / etc
#NeurIPS2024
32-72B is all we need for human level of AGI 🤖
Anything higher is just us being inefficient in architecture / code / etc
#NeurIPS2024
Reposted by PicoCreator - AI Model Builder 🛫 NeurIPS
Lots of people are only looking out for themselves which is why they fear everything.
December 11, 2024 at 4:09 PM
Lots of people are only looking out for themselves which is why they fear everything.
PS: once we get 70B converted and working at ~128k context length
We will be able to cover the vast majority of enterprise AI workloads without QKV attention
Let that sink in 😎
We will be able to cover the vast majority of enterprise AI workloads without QKV attention
Let that sink in 😎
QKV Attention is **not** all you need
We release QRWKV6-32B-Instruct preview, a model converted from Qwen-32B instruct, trained for several hours on 2 MI300 nodes.
Surpassing all previous known open linear models (StateSpace, Hybrid, etc)
Unlocking 1000x+ lower inference cost
We release QRWKV6-32B-Instruct preview, a model converted from Qwen-32B instruct, trained for several hours on 2 MI300 nodes.
Surpassing all previous known open linear models (StateSpace, Hybrid, etc)
Unlocking 1000x+ lower inference cost
December 11, 2024 at 9:22 PM
PS: once we get 70B converted and working at ~128k context length
We will be able to cover the vast majority of enterprise AI workloads without QKV attention
Let that sink in 😎
We will be able to cover the vast majority of enterprise AI workloads without QKV attention
Let that sink in 😎
QKV Attention is **not** all you need
We release QRWKV6-32B-Instruct preview, a model converted from Qwen-32B instruct, trained for several hours on 2 MI300 nodes.
Surpassing all previous known open linear models (StateSpace, Hybrid, etc)
Unlocking 1000x+ lower inference cost
We release QRWKV6-32B-Instruct preview, a model converted from Qwen-32B instruct, trained for several hours on 2 MI300 nodes.
Surpassing all previous known open linear models (StateSpace, Hybrid, etc)
Unlocking 1000x+ lower inference cost
December 11, 2024 at 9:17 PM
QKV Attention is **not** all you need
We release QRWKV6-32B-Instruct preview, a model converted from Qwen-32B instruct, trained for several hours on 2 MI300 nodes.
Surpassing all previous known open linear models (StateSpace, Hybrid, etc)
Unlocking 1000x+ lower inference cost
We release QRWKV6-32B-Instruct preview, a model converted from Qwen-32B instruct, trained for several hours on 2 MI300 nodes.
Surpassing all previous known open linear models (StateSpace, Hybrid, etc)
Unlocking 1000x+ lower inference cost
So yea, we just finished up the best subquadratic model for our QRWKV varient. Landing in hot during neurips
Matching transformer level performance despite the lack of "Quadratic Attention", using RWKV Attention instead
Proving Attention is **not** all you need
Matching transformer level performance despite the lack of "Quadratic Attention", using RWKV Attention instead
Proving Attention is **not** all you need
December 11, 2024 at 11:14 AM
So yea, we just finished up the best subquadratic model for our QRWKV varient. Landing in hot during neurips
Matching transformer level performance despite the lack of "Quadratic Attention", using RWKV Attention instead
Proving Attention is **not** all you need
Matching transformer level performance despite the lack of "Quadratic Attention", using RWKV Attention instead
Proving Attention is **not** all you need
Did the RNG bless your NeurIPS lottery?
Are you in SF? Wish to nerd out on AI & ML?
Join our holiday get together + community potluck + discussion of the best new AI research with me,
@swyx.io @eraqian @dylan522p @vibhuuuus 😎
At 4pm today
lu.ma/25mwbwcm
Are you in SF? Wish to nerd out on AI & ML?
Join our holiday get together + community potluck + discussion of the best new AI research with me,
@swyx.io @eraqian @dylan522p @vibhuuuus 😎
At 4pm today
lu.ma/25mwbwcm
NeurIPS Pre-Game & Holiday Potluck · Luma
Endless papers, so little time—let’s prep for NeurIPS together! 📚✨
With the big week just around the corner, take a break from solo paper crunching and join…
lu.ma
December 8, 2024 at 6:35 PM
Did the RNG bless your NeurIPS lottery?
Are you in SF? Wish to nerd out on AI & ML?
Join our holiday get together + community potluck + discussion of the best new AI research with me,
@swyx.io @eraqian @dylan522p @vibhuuuus 😎
At 4pm today
lu.ma/25mwbwcm
Are you in SF? Wish to nerd out on AI & ML?
Join our holiday get together + community potluck + discussion of the best new AI research with me,
@swyx.io @eraqian @dylan522p @vibhuuuus 😎
At 4pm today
lu.ma/25mwbwcm
Did you hear that joke about Ice Cream?
Best I keep that frozen for awhile!
Best I keep that frozen for awhile!
Did you hear the one about honey?
I’ll just bee quiet about it!
I’ll just bee quiet about it!
You hear that one joke about jam?
Best I keep that one preserved
Best I keep that one preserved
December 8, 2024 at 5:56 AM
Did you hear that joke about Ice Cream?
Best I keep that frozen for awhile!
Best I keep that frozen for awhile!
Reposted by PicoCreator - AI Model Builder 🛫 NeurIPS
Repeat after me:
I will build evals for my tasks.
I will build evals for my tasks.
I will build evals for my tasks.
I will build evals for my tasks.
I will build evals for my tasks.
I will build evals for my tasks.
December 7, 2024 at 5:55 PM
Repeat after me:
I will build evals for my tasks.
I will build evals for my tasks.
I will build evals for my tasks.
I will build evals for my tasks.
I will build evals for my tasks.
I will build evals for my tasks.
Reposted by PicoCreator - AI Model Builder 🛫 NeurIPS
This is such an interesting angle, because the number of large organizations that have this level of discipline to evaluating new models is likely to be tiny
Running a bunch of test prompts specific to what your company does through a new model feels like it should be pretty low hanging fruit
Running a bunch of test prompts specific to what your company does through a new model feels like it should be pretty low hanging fruit
A test of how seriously your firm is taking AI: when o-1 (& the new Gemini model) came out this week, were there assigned folks who immediately ran the model through your internal, validated, firm-specific benchmarks to see how useful it as? Did you update any plans or goals as a result?
December 7, 2024 at 5:00 PM
This is such an interesting angle, because the number of large organizations that have this level of discipline to evaluating new models is likely to be tiny
Running a bunch of test prompts specific to what your company does through a new model feels like it should be pretty low hanging fruit
Running a bunch of test prompts specific to what your company does through a new model feels like it should be pretty low hanging fruit
Reposted by PicoCreator - AI Model Builder 🛫 NeurIPS
A test of how seriously your firm is taking AI: when o-1 (& the new Gemini model) came out this week, were there assigned folks who immediately ran the model through your internal, validated, firm-specific benchmarks to see how useful it as? Did you update any plans or goals as a result?
December 7, 2024 at 4:34 PM
A test of how seriously your firm is taking AI: when o-1 (& the new Gemini model) came out this week, were there assigned folks who immediately ran the model through your internal, validated, firm-specific benchmarks to see how useful it as? Did you update any plans or goals as a result?
Reposted by PicoCreator - AI Model Builder 🛫 NeurIPS
I'm heading to #NeurIPS next week, I'll be in town Wednesday-Friday!
I'll be at a couple things:
- Wed 1-2pm: talking Transformer killers with
@picocreator.bsky.social at @swyx.io @latentspacepod.bsky.social live!
- Wed 11am: RedPajama poster (spotlight) with
Maurice Weber
1/2
I'll be at a couple things:
- Wed 1-2pm: talking Transformer killers with
@picocreator.bsky.social at @swyx.io @latentspacepod.bsky.social live!
- Wed 11am: RedPajama poster (spotlight) with
Maurice Weber
1/2
December 6, 2024 at 1:40 AM
I'm heading to #NeurIPS next week, I'll be in town Wednesday-Friday!
I'll be at a couple things:
- Wed 1-2pm: talking Transformer killers with
@picocreator.bsky.social at @swyx.io @latentspacepod.bsky.social live!
- Wed 11am: RedPajama poster (spotlight) with
Maurice Weber
1/2
I'll be at a couple things:
- Wed 1-2pm: talking Transformer killers with
@picocreator.bsky.social at @swyx.io @latentspacepod.bsky.social live!
- Wed 11am: RedPajama poster (spotlight) with
Maurice Weber
1/2
Just landed in Toronto 🇨🇦
Me as South East Asian: oooo… snow ☃️
My Canadian friends: that’s barely any snow ❄️
The true Canadian experience needs to have at least knee high snow I guess 🤣
Me as South East Asian: oooo… snow ☃️
My Canadian friends: that’s barely any snow ❄️
The true Canadian experience needs to have at least knee high snow I guess 🤣
December 5, 2024 at 10:12 PM
Just landed in Toronto 🇨🇦
Me as South East Asian: oooo… snow ☃️
My Canadian friends: that’s barely any snow ❄️
The true Canadian experience needs to have at least knee high snow I guess 🤣
Me as South East Asian: oooo… snow ☃️
My Canadian friends: that’s barely any snow ❄️
The true Canadian experience needs to have at least knee high snow I guess 🤣
For those in SF 🌉 before Neurips (8th Dec), doing a casual pre-neurips gathering; Food, Friends, Papers.
To chill and talk, before flying over to Canada 🇨🇦
lu.ma/25mwbwcm
To chill and talk, before flying over to Canada 🇨🇦
lu.ma/25mwbwcm
NeurIPS Pre-Game & Holiday Potlock · Luma
Endless papers, so little time—let’s prep for NeurIPS together! 📚✨
With the big week just around the corner, take a break from solo paper crunching and join…
lu.ma
December 5, 2024 at 3:23 AM
For those in SF 🌉 before Neurips (8th Dec), doing a casual pre-neurips gathering; Food, Friends, Papers.
To chill and talk, before flying over to Canada 🇨🇦
lu.ma/25mwbwcm
To chill and talk, before flying over to Canada 🇨🇦
lu.ma/25mwbwcm
So what does one do, if they use more then 1TB of storage on huggingface?
Seems like pro (~$9/month) is limited to 1TB?
Seems like pro (~$9/month) is limited to 1TB?
December 2, 2024 at 11:47 PM
So what does one do, if they use more then 1TB of storage on huggingface?
Seems like pro (~$9/month) is limited to 1TB?
Seems like pro (~$9/month) is limited to 1TB?
So i heard ChatGPT is looking into adding ads now ...
( the circle of internet life )
( the circle of internet life )
December 2, 2024 at 11:27 PM
So i heard ChatGPT is looking into adding ads now ...
( the circle of internet life )
( the circle of internet life )
Looking at what Qwen and Deepseek achieved at way less then 1/10th the budget (or even 1/100th)
The biggest damage big AI did to western AI - was to convince everyone that, only they can build AI with more funding - and that no one else can
Denying competition and progress locally
The biggest damage big AI did to western AI - was to convince everyone that, only they can build AI with more funding - and that no one else can
Denying competition and progress locally
November 29, 2024 at 10:11 AM
Looking at what Qwen and Deepseek achieved at way less then 1/10th the budget (or even 1/100th)
The biggest damage big AI did to western AI - was to convince everyone that, only they can build AI with more funding - and that no one else can
Denying competition and progress locally
The biggest damage big AI did to western AI - was to convince everyone that, only they can build AI with more funding - and that no one else can
Denying competition and progress locally
Reposted by PicoCreator - AI Model Builder 🛫 NeurIPS
and guests @alpindale.bsky.social @capetorch.bsky.social and @picocreator.bsky.social for a great show today!
Give them all a follow folks 👆
P.S - A shoutout to @presidentlin.bsky.social for helping as always
Give them all a follow folks 👆
P.S - A shoutout to @presidentlin.bsky.social for helping as always
November 29, 2024 at 12:16 AM
and guests @alpindale.bsky.social @capetorch.bsky.social and @picocreator.bsky.social for a great show today!
Give them all a follow folks 👆
P.S - A shoutout to @presidentlin.bsky.social for helping as always
Give them all a follow folks 👆
P.S - A shoutout to @presidentlin.bsky.social for helping as always
Reposted by PicoCreator - AI Model Builder 🛫 NeurIPS
Over the past 18 month, the @latentspacepod.bsky.social paper club has had an unbroken streak of hosting EVERY SINGLE WEEK. We gained technical knowledge + insider know-how, built friendships, and grown a community of learners.
Here's how to start your own paper club
eugeneyan.com/writing/pape...
Here's how to start your own paper club
eugeneyan.com/writing/pape...
How to Run a Weekly Paper Club (and Build a Learning Community)
Benefits of running a weekly paper club, how to start one, and how to read and facilitate papers.
eugeneyan.com
November 27, 2024 at 12:42 AM
Over the past 18 month, the @latentspacepod.bsky.social paper club has had an unbroken streak of hosting EVERY SINGLE WEEK. We gained technical knowledge + insider know-how, built friendships, and grown a community of learners.
Here's how to start your own paper club
eugeneyan.com/writing/pape...
Here's how to start your own paper club
eugeneyan.com/writing/pape...
@eugeneyan.bsky.social
Turning this into a perpetual thread.... for every time i hear
"LLM-as-a-judge" or something "similar" being discussed
Today Oct 30 : Heavybit devguild AI Summit 3 event
Turning this into a perpetual thread.... for every time i hear
"LLM-as-a-judge" or something "similar" being discussed
Today Oct 30 : Heavybit devguild AI Summit 3 event
October 30, 2024 at 11:45 PM
@eugeneyan.bsky.social
Turning this into a perpetual thread.... for every time i hear
"LLM-as-a-judge" or something "similar" being discussed
Today Oct 30 : Heavybit devguild AI Summit 3 event
Turning this into a perpetual thread.... for every time i hear
"LLM-as-a-judge" or something "similar" being discussed
Today Oct 30 : Heavybit devguild AI Summit 3 event
Hello bluesky, my twitter is : x.com/picocreator
And I guess I should start cross posting here as well
And I guess I should start cross posting here as well
x.com
x.com
October 30, 2024 at 8:25 PM
Hello bluesky, my twitter is : x.com/picocreator
And I guess I should start cross posting here as well
And I guess I should start cross posting here as well