Shelby Heinecke
@shelbyhai.bsky.social
Leading AI Research & Innovation.
PhD in ML Theory. Math BS @MIT.
Posting about AI, leadership, and growth.
Let’s build the future together. 🚀
PhD in ML Theory. Math BS @MIT.
Posting about AI, leadership, and growth.
Let’s build the future together. 🚀
Pinned
Shelby Heinecke
@shelbyhai.bsky.social
· Nov 19
The future of AI will be hybrid: some models on the cloud, some on-device.
On-device models give you privacy, offline availability, lower latency, and lower cost-to-serve.
Some tasks are simply better local.
On-device models give you privacy, offline availability, lower latency, and lower cost-to-serve.
Some tasks are simply better local.
My new website just dropped: shelbyh.ai/
I wanted one place to share my work on enterprise AI, leadership, and research. More coming soon.
Built with @framer - the best website building platform I've used so far
I wanted one place to share my work on enterprise AI, leadership, and research. More coming soon.
Built with @framer - the best website building platform I've used so far
Shelby Heinecke, PhD — Enterprise AI Research & Leadership
Shelby Heinecke, PhD is an enterprise AI research leader and speaker bridging cutting-edge AI research, product innovation, and leadership to drive real-world impact.
shelbyh.ai
December 18, 2025 at 5:27 PM
My new website just dropped: shelbyh.ai/
I wanted one place to share my work on enterprise AI, leadership, and research. More coming soon.
Built with @framer - the best website building platform I've used so far
I wanted one place to share my work on enterprise AI, leadership, and research. More coming soon.
Built with @framer - the best website building platform I've used so far
For agentic tasks, trajectory data is everything.
A trajectory = (task, action sequence).
Show a model thousands of these, and it becomes extremely good at acting in that domain.
LLMs don’t learn actions abstractly.
They learn from demonstrations.
A trajectory = (task, action sequence).
Show a model thousands of these, and it becomes extremely good at acting in that domain.
LLMs don’t learn actions abstractly.
They learn from demonstrations.
December 18, 2025 at 4:00 PM
For agentic tasks, trajectory data is everything.
A trajectory = (task, action sequence).
Show a model thousands of these, and it becomes extremely good at acting in that domain.
LLMs don’t learn actions abstractly.
They learn from demonstrations.
A trajectory = (task, action sequence).
Show a model thousands of these, and it becomes extremely good at acting in that domain.
LLMs don’t learn actions abstractly.
They learn from demonstrations.
Smart task routing is going to be one of the biggest cost reductions in enterprise AI:
simple queries → small local model
complex reasoning → larger model only when necessary
This hybrid architecture is the future.
simple queries → small local model
complex reasoning → larger model only when necessary
This hybrid architecture is the future.
December 12, 2025 at 3:58 PM
Smart task routing is going to be one of the biggest cost reductions in enterprise AI:
simple queries → small local model
complex reasoning → larger model only when necessary
This hybrid architecture is the future.
simple queries → small local model
complex reasoning → larger model only when necessary
This hybrid architecture is the future.
LLMs can generate synthetic data, but the real magic is in validation.
Layered checks like:
‣ trajectory correctness
‣ function-call execution
‣ goal completion
‣ instruction alignment
Anything that fails gets discarded or corrected.
A good synthetic dataset is mostly filtration.
Layered checks like:
‣ trajectory correctness
‣ function-call execution
‣ goal completion
‣ instruction alignment
Anything that fails gets discarded or corrected.
A good synthetic dataset is mostly filtration.
December 11, 2025 at 3:59 PM
LLMs can generate synthetic data, but the real magic is in validation.
Layered checks like:
‣ trajectory correctness
‣ function-call execution
‣ goal completion
‣ instruction alignment
Anything that fails gets discarded or corrected.
A good synthetic dataset is mostly filtration.
Layered checks like:
‣ trajectory correctness
‣ function-call execution
‣ goal completion
‣ instruction alignment
Anything that fails gets discarded or corrected.
A good synthetic dataset is mostly filtration.
Wild to see how far local LMs (<20B) have come in just two years.
2023 → barely usable for anything beyond simple tasks.
2025 → crushing intermediate chat tasks, and even doing surprisingly well in difficult chats.
Local AI is heating up 🔥
2023 → barely usable for anything beyond simple tasks.
2025 → crushing intermediate chat tasks, and even doing surprisingly well in difficult chats.
Local AI is heating up 🔥
December 5, 2025 at 4:04 PM
Wild to see how far local LMs (<20B) have come in just two years.
2023 → barely usable for anything beyond simple tasks.
2025 → crushing intermediate chat tasks, and even doing surprisingly well in difficult chats.
Local AI is heating up 🔥
2023 → barely usable for anything beyond simple tasks.
2025 → crushing intermediate chat tasks, and even doing surprisingly well in difficult chats.
Local AI is heating up 🔥
Give a small model targeted, verified demonstrations
‣ actions
‣ reasoning steps
‣ navigation patterns
and suddenly it can match or beat much larger models trained on generic data.
The magic isn’t the model size.
It’s the data pipeline.
‣ actions
‣ reasoning steps
‣ navigation patterns
and suddenly it can match or beat much larger models trained on generic data.
The magic isn’t the model size.
It’s the data pipeline.
December 5, 2025 at 12:01 AM
Give a small model targeted, verified demonstrations
‣ actions
‣ reasoning steps
‣ navigation patterns
and suddenly it can match or beat much larger models trained on generic data.
The magic isn’t the model size.
It’s the data pipeline.
‣ actions
‣ reasoning steps
‣ navigation patterns
and suddenly it can match or beat much larger models trained on generic data.
The magic isn’t the model size.
It’s the data pipeline.
Small language models aren’t for everything.
But for narrow, well-defined tasks, they’re pretty solid.
But for narrow, well-defined tasks, they’re pretty solid.
December 3, 2025 at 3:59 PM
Small language models aren’t for everything.
But for narrow, well-defined tasks, they’re pretty solid.
But for narrow, well-defined tasks, they’re pretty solid.
How to get mentors:
• Pick people you want to learn from
• Ask 1–2 specific questions
• Follow up
• Let the relationship evolve
That’s it. No formalities required.
• Pick people you want to learn from
• Ask 1–2 specific questions
• Follow up
• Let the relationship evolve
That’s it. No formalities required.
December 1, 2025 at 3:59 PM
How to get mentors:
• Pick people you want to learn from
• Ask 1–2 specific questions
• Follow up
• Let the relationship evolve
That’s it. No formalities required.
• Pick people you want to learn from
• Ask 1–2 specific questions
• Follow up
• Let the relationship evolve
That’s it. No formalities required.
More and more I’m realizing we’re not “finding” ourselves, we’re revealing ourselves.
Everything is already inside us.
Experience just brings it into focus.
Everything is already inside us.
Experience just brings it into focus.
November 29, 2025 at 10:10 PM
More and more I’m realizing we’re not “finding” ourselves, we’re revealing ourselves.
Everything is already inside us.
Experience just brings it into focus.
Everything is already inside us.
Experience just brings it into focus.
So where is everyone headed next week - #NeurIPS2025 or #AWSreInvent?
November 28, 2025 at 9:49 PM
So where is everyone headed next week - #NeurIPS2025 or #AWSreInvent?
Happy Thanksgiving Everyone! 🦃 ✨
November 27, 2025 at 8:02 PM
Happy Thanksgiving Everyone! 🦃 ✨
If you don’t have a brag doc, you’re already behind.
Track your contributions and the impact they created.
Your future self (and your promo case) will thank you.
Track your contributions and the impact they created.
Your future self (and your promo case) will thank you.
November 26, 2025 at 4:01 PM
If you don’t have a brag doc, you’re already behind.
Track your contributions and the impact they created.
Your future self (and your promo case) will thank you.
Track your contributions and the impact they created.
Your future self (and your promo case) will thank you.
On-device AI = your data stays on your device.
For privacy-sensitive use cases, that’s a game-changer.
For privacy-sensitive use cases, that’s a game-changer.
November 25, 2025 at 4:03 PM
On-device AI = your data stays on your device.
For privacy-sensitive use cases, that’s a game-changer.
For privacy-sensitive use cases, that’s a game-changer.
When stuck in a meeting, remember this: every meeting is a mini-stage.
Use them as reps for public speaking, teaching, and leading.
Use them as reps for public speaking, teaching, and leading.
November 24, 2025 at 11:57 PM
When stuck in a meeting, remember this: every meeting is a mini-stage.
Use them as reps for public speaking, teaching, and leading.
Use them as reps for public speaking, teaching, and leading.
If you’re waiting for mentorship, you’re waiting too long.
Start the conversation.
Start the conversation.
November 24, 2025 at 4:02 PM
If you’re waiting for mentorship, you’re waiting too long.
Start the conversation.
Start the conversation.
One of my favorite things about Sundays:
New episode of Welcome to Derry 🎈
New episode of Welcome to Derry 🎈
November 24, 2025 at 1:03 AM
One of my favorite things about Sundays:
New episode of Welcome to Derry 🎈
New episode of Welcome to Derry 🎈
If your leadership stops at tracking team tasks, something’s missing.
Managing tasks is <25% of the job.
People, technical clarity, and alignment are the rest.
Managing tasks is <25% of the job.
People, technical clarity, and alignment are the rest.
November 24, 2025 at 12:03 AM
If your leadership stops at tracking team tasks, something’s missing.
Managing tasks is <25% of the job.
People, technical clarity, and alignment are the rest.
Managing tasks is <25% of the job.
People, technical clarity, and alignment are the rest.
We want AI that “knows us.”
But our data? Fragmented, noisy, inconsistent.
Introducing PersonaBench, a synthetic, human-like dataset (bios, interests, conversations, purchases) to test how well today's RAG systems handle real user complexity.
But our data? Fragmented, noisy, inconsistent.
Introducing PersonaBench, a synthetic, human-like dataset (bios, interests, conversations, purchases) to test how well today's RAG systems handle real user complexity.
November 23, 2025 at 3:59 PM
We want AI that “knows us.”
But our data? Fragmented, noisy, inconsistent.
Introducing PersonaBench, a synthetic, human-like dataset (bios, interests, conversations, purchases) to test how well today's RAG systems handle real user complexity.
But our data? Fragmented, noisy, inconsistent.
Introducing PersonaBench, a synthetic, human-like dataset (bios, interests, conversations, purchases) to test how well today's RAG systems handle real user complexity.
Technical skills have levels.
Leadership does too.
Go deeper and it becomes all about clarity, influence, and your impact on people.
Leadership does too.
Go deeper and it becomes all about clarity, influence, and your impact on people.
November 22, 2025 at 4:03 PM
Technical skills have levels.
Leadership does too.
Go deeper and it becomes all about clarity, influence, and your impact on people.
Leadership does too.
Go deeper and it becomes all about clarity, influence, and your impact on people.
With MCP, your agent has access to tools. Great!
But that doesn't mean your agent knows how to use those tools.
Tool access ≠ Tool use ability
That's why we created MCPEval (#EMNLP2025), a new framework for evaluating agent performance on any MCP server.
But that doesn't mean your agent knows how to use those tools.
Tool access ≠ Tool use ability
That's why we created MCPEval (#EMNLP2025), a new framework for evaluating agent performance on any MCP server.
November 21, 2025 at 11:58 PM
With MCP, your agent has access to tools. Great!
But that doesn't mean your agent knows how to use those tools.
Tool access ≠ Tool use ability
That's why we created MCPEval (#EMNLP2025), a new framework for evaluating agent performance on any MCP server.
But that doesn't mean your agent knows how to use those tools.
Tool access ≠ Tool use ability
That's why we created MCPEval (#EMNLP2025), a new framework for evaluating agent performance on any MCP server.
Developing an AI model is only one step on the road to enterprise deployment.
The rest is where things get tricky:
The rest is where things get tricky:
November 21, 2025 at 3:56 PM
Developing an AI model is only one step on the road to enterprise deployment.
The rest is where things get tricky:
The rest is where things get tricky:
Presented your work and got silence?
No questions. No reactions. Just blank stares.
Try this:
• Talk slower
• Pause for questions
• Skimmable slides
• Tailor to the audience
No questions. No reactions. Just blank stares.
Try this:
• Talk slower
• Pause for questions
• Skimmable slides
• Tailor to the audience
November 21, 2025 at 1:04 AM
Presented your work and got silence?
No questions. No reactions. Just blank stares.
Try this:
• Talk slower
• Pause for questions
• Skimmable slides
• Tailor to the audience
No questions. No reactions. Just blank stares.
Try this:
• Talk slower
• Pause for questions
• Skimmable slides
• Tailor to the audience
If I were breaking into AI today, here’s exactly where I’d start:
(and no, you don’t need a PhD)
• Learn the bleeding-edge at NeurIPS, ICLR, ICML
• Study how companies deploy AI at places like The AI Conference, Ai4, MLOps
• Implement research papers
• Find or start reading groups
(and no, you don’t need a PhD)
• Learn the bleeding-edge at NeurIPS, ICLR, ICML
• Study how companies deploy AI at places like The AI Conference, Ai4, MLOps
• Implement research papers
• Find or start reading groups
November 20, 2025 at 3:57 PM
If I were breaking into AI today, here’s exactly where I’d start:
(and no, you don’t need a PhD)
• Learn the bleeding-edge at NeurIPS, ICLR, ICML
• Study how companies deploy AI at places like The AI Conference, Ai4, MLOps
• Implement research papers
• Find or start reading groups
(and no, you don’t need a PhD)
• Learn the bleeding-edge at NeurIPS, ICLR, ICML
• Study how companies deploy AI at places like The AI Conference, Ai4, MLOps
• Implement research papers
• Find or start reading groups
Recently spoke about our small language models at Motherduck's Small Data conference.
I have to say - they have the best swag 🔥
I have to say - they have the best swag 🔥
November 20, 2025 at 4:04 AM
Recently spoke about our small language models at Motherduck's Small Data conference.
I have to say - they have the best swag 🔥
I have to say - they have the best swag 🔥
The future of AI will be hybrid: some models on the cloud, some on-device.
On-device models give you privacy, offline availability, lower latency, and lower cost-to-serve.
Some tasks are simply better local.
On-device models give you privacy, offline availability, lower latency, and lower cost-to-serve.
Some tasks are simply better local.
November 19, 2025 at 3:57 PM
The future of AI will be hybrid: some models on the cloud, some on-device.
On-device models give you privacy, offline availability, lower latency, and lower cost-to-serve.
Some tasks are simply better local.
On-device models give you privacy, offline availability, lower latency, and lower cost-to-serve.
Some tasks are simply better local.