Omar Khattab
@lateinteraction.bsky.social
Incoming asst professor at MIT EECS, Fall 2025. Research scientist at Databricks. CS PhD @StanfordNLP.bsky.social. Author of ColBERT.ai & DSPy.ai.
Reposted by Omar Khattab
Drew’s post is well worth reading as DSPy seems to be a missing link in thinking about LLM usage. Very readable and interesting. www.dbreunig.com/2025/06/10/l...
Thank you @simonwillison.net
Thank you @simonwillison.net
October 6, 2025 at 11:42 AM
Drew’s post is well worth reading as DSPy seems to be a missing link in thinking about LLM usage. Very readable and interesting. www.dbreunig.com/2025/06/10/l...
Thank you @simonwillison.net
Thank you @simonwillison.net
Reposted by Omar Khattab
If you've been trying to figure out DSPy - the automatic prompt optimization system - this talk by @dbreunig.bsky.social is the clearest explanation I've seen yet, with a very useful real-world case study www.youtube.com/watch?v=I9Zt...
My notes here: simonwillison.net/2025/Oct/4/d...
My notes here: simonwillison.net/2025/Oct/4/d...
Let the LLM Write the Prompts: An Intro to DSPy in Compound AI Pipelines
YouTube video by Databricks
www.youtube.com
October 4, 2025 at 11:05 PM
If you've been trying to figure out DSPy - the automatic prompt optimization system - this talk by @dbreunig.bsky.social is the clearest explanation I've seen yet, with a very useful real-world case study www.youtube.com/watch?v=I9Zt...
My notes here: simonwillison.net/2025/Oct/4/d...
My notes here: simonwillison.net/2025/Oct/4/d...
Reposted by Omar Khattab
#pydatabos interesting! How the Arbor library works under the hood hand in hand with DSPy
October 15, 2025 at 11:42 PM
#pydatabos interesting! How the Arbor library works under the hood hand in hand with DSPy
premature optimization is the sqrt of all evil
October 29, 2025 at 4:26 PM
premature optimization is the sqrt of all evil
Reposted by Omar Khattab
#pydatabos one line motivation for using DSPy!
October 15, 2025 at 11:56 PM
#pydatabos one line motivation for using DSPy!
Reposted by Omar Khattab
Stop what you are doing and try out GEPA now!
"GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning" presents such elegant ideas by a collection of amazing researchers!
Here is a tldr of how it works:
"GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning" presents such elegant ideas by a collection of amazing researchers!
Here is a tldr of how it works:
October 21, 2025 at 3:03 PM
Stop what you are doing and try out GEPA now!
"GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning" presents such elegant ideas by a collection of amazing researchers!
Here is a tldr of how it works:
"GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning" presents such elegant ideas by a collection of amazing researchers!
Here is a tldr of how it works:
Reposted by Omar Khattab
colbert-muvera-micro a 4M(!!) late interaction model
late interaction models do embedding vector index queries and reranking at the same time leading to far higher accuracy
huggingface.co/NeuML/colber...
late interaction models do embedding vector index queries and reranking at the same time leading to far higher accuracy
huggingface.co/NeuML/colber...
September 19, 2025 at 11:15 AM
colbert-muvera-micro a 4M(!!) late interaction model
late interaction models do embedding vector index queries and reranking at the same time leading to far higher accuracy
huggingface.co/NeuML/colber...
late interaction models do embedding vector index queries and reranking at the same time leading to far higher accuracy
huggingface.co/NeuML/colber...
Reposted by Omar Khattab
Reposted by Omar Khattab
Here's the write up of my Data+AI Summit talk on the perils of prompts in code and how to mitigate them with DSPy. www.dbreunig.com/2025/06/10/l...
Let the Model Write the Prompt
Notes from a talk I delivered at the 2025 Data + AI Summit, detailing the problem with prompts in your code and how DSPy can make everything better.
www.dbreunig.com
June 15, 2025 at 4:57 PM
Here's the write up of my Data+AI Summit talk on the perils of prompts in code and how to mitigate them with DSPy. www.dbreunig.com/2025/06/10/l...
Reposted by Omar Khattab
📣 TODAY at 4PM PT - MLflow Community Meetup!
🔗 Register today 👉 lu.ma/mlflow423
Join the global MLflow community for two exciting tech deep dives:
🔹 MLflow + #DSPy Integration
🔹 Cleanlab + #MLflow
🎥 Streaming live on YouTube, LinkedIn, and X
💬 Live Q&A with the presenters
#opensource #oss
🔗 Register today 👉 lu.ma/mlflow423
Join the global MLflow community for two exciting tech deep dives:
🔹 MLflow + #DSPy Integration
🔹 Cleanlab + #MLflow
🎥 Streaming live on YouTube, LinkedIn, and X
💬 Live Q&A with the presenters
#opensource #oss
MLflow Community Meetup | April 23 · Luma
Join us for the next MLflow Community Meetup — Wednesday, April 23 at 4PM PT!
We’re bringing two exciting presentations to the community:
🔹 MLflow + DSPy…
lu.ma
April 23, 2025 at 7:21 PM
📣 TODAY at 4PM PT - MLflow Community Meetup!
🔗 Register today 👉 lu.ma/mlflow423
Join the global MLflow community for two exciting tech deep dives:
🔹 MLflow + #DSPy Integration
🔹 Cleanlab + #MLflow
🎥 Streaming live on YouTube, LinkedIn, and X
💬 Live Q&A with the presenters
#opensource #oss
🔗 Register today 👉 lu.ma/mlflow423
Join the global MLflow community for two exciting tech deep dives:
🔹 MLflow + #DSPy Integration
🔹 Cleanlab + #MLflow
🎥 Streaming live on YouTube, LinkedIn, and X
💬 Live Q&A with the presenters
#opensource #oss
Reposted by Omar Khattab
MLflow now supports tracking for #DSPy (Community) optimization — just like it does for @pytorch.org training! 🙌
#MLflow is the first to bring full visibility into DSPy’s prompt optimization process. More observability, less guesswork.
Get started today! ➡️ medium.com/@AI-on-Datab...
#opensource
#MLflow is the first to bring full visibility into DSPy’s prompt optimization process. More observability, less guesswork.
Get started today! ➡️ medium.com/@AI-on-Datab...
#opensource
April 21, 2025 at 7:20 PM
MLflow now supports tracking for #DSPy (Community) optimization — just like it does for @pytorch.org training! 🙌
#MLflow is the first to bring full visibility into DSPy’s prompt optimization process. More observability, less guesswork.
Get started today! ➡️ medium.com/@AI-on-Datab...
#opensource
#MLflow is the first to bring full visibility into DSPy’s prompt optimization process. More observability, less guesswork.
Get started today! ➡️ medium.com/@AI-on-Datab...
#opensource
Reposted by Omar Khattab
Join us for the next MLflow Community Meetup — Wednesday, April 23 at 4PM PT! 🗓️
🔹 Explore the new MLflow + #DSPy integration
🔹 Learn how Cleanlab adds trust to AI workflows with MLflow
💬 Live Q&A + demos
📺 Streamed on YouTube, LinkedIn, and X
👉 RSVP: lu.ma/mlflow423
#opensource #mlflow #oss
🔹 Explore the new MLflow + #DSPy integration
🔹 Learn how Cleanlab adds trust to AI workflows with MLflow
💬 Live Q&A + demos
📺 Streamed on YouTube, LinkedIn, and X
👉 RSVP: lu.ma/mlflow423
#opensource #mlflow #oss
MLflow Monthly Meetup · Luma
Join us for the next MLflow Community Meetup — Wednesday, April 23 at 4PM PT!
We’re bringing two exciting presentations to the community:
🔹 MLflow + DSPy…
lu.ma
April 15, 2025 at 7:51 PM
Join us for the next MLflow Community Meetup — Wednesday, April 23 at 4PM PT! 🗓️
🔹 Explore the new MLflow + #DSPy integration
🔹 Learn how Cleanlab adds trust to AI workflows with MLflow
💬 Live Q&A + demos
📺 Streamed on YouTube, LinkedIn, and X
👉 RSVP: lu.ma/mlflow423
#opensource #mlflow #oss
🔹 Explore the new MLflow + #DSPy integration
🔹 Learn how Cleanlab adds trust to AI workflows with MLflow
💬 Live Q&A + demos
📺 Streamed on YouTube, LinkedIn, and X
👉 RSVP: lu.ma/mlflow423
#opensource #mlflow #oss
Yes there's an evals crisis, but evaluating *models* is not even the right question most of the time
LangProBe from Shangyin Tan, @lakshyaaagrawal.bsky.social, Arnav Singhvi, Liheng Lai, @michaelryan207.bsky.social et al begins to ask what complete *AI systems* we should build & under what settings
LangProBe from Shangyin Tan, @lakshyaaagrawal.bsky.social, Arnav Singhvi, Liheng Lai, @michaelryan207.bsky.social et al begins to ask what complete *AI systems* we should build & under what settings
🧵Introducing LangProBe: the first benchmark testing where and how composing LLMs into language programs affects cost-quality tradeoffs!
We find that, on avg across diverse tasks, smaller models within optimized programs beat calls to larger models at a fraction of the cost.
We find that, on avg across diverse tasks, smaller models within optimized programs beat calls to larger models at a fraction of the cost.
March 3, 2025 at 7:42 PM
Yes there's an evals crisis, but evaluating *models* is not even the right question most of the time
LangProBe from Shangyin Tan, @lakshyaaagrawal.bsky.social, Arnav Singhvi, Liheng Lai, @michaelryan207.bsky.social et al begins to ask what complete *AI systems* we should build & under what settings
LangProBe from Shangyin Tan, @lakshyaaagrawal.bsky.social, Arnav Singhvi, Liheng Lai, @michaelryan207.bsky.social et al begins to ask what complete *AI systems* we should build & under what settings
Reposted by Omar Khattab
🧵Introducing LangProBe: the first benchmark testing where and how composing LLMs into language programs affects cost-quality tradeoffs!
We find that, on avg across diverse tasks, smaller models within optimized programs beat calls to larger models at a fraction of the cost.
We find that, on avg across diverse tasks, smaller models within optimized programs beat calls to larger models at a fraction of the cost.
March 3, 2025 at 6:59 PM
🧵Introducing LangProBe: the first benchmark testing where and how composing LLMs into language programs affects cost-quality tradeoffs!
We find that, on avg across diverse tasks, smaller models within optimized programs beat calls to larger models at a fraction of the cost.
We find that, on avg across diverse tasks, smaller models within optimized programs beat calls to larger models at a fraction of the cost.
Composition & abstraction are the foundations of CS, but are clearly absent in modern ML.
It's not that they're not crucial for intelligent software. But it takes building many half-working systems to abstract successfully, and it takes good abstractions to have primitives worth composing.
🧵1/2
It's not that they're not crucial for intelligent software. But it takes building many half-working systems to abstract successfully, and it takes good abstractions to have primitives worth composing.
🧵1/2
February 26, 2025 at 10:00 PM
Composition & abstraction are the foundations of CS, but are clearly absent in modern ML.
It's not that they're not crucial for intelligent software. But it takes building many half-working systems to abstract successfully, and it takes good abstractions to have primitives worth composing.
🧵1/2
It's not that they're not crucial for intelligent software. But it takes building many half-working systems to abstract successfully, and it takes good abstractions to have primitives worth composing.
🧵1/2
Some quick thoughts: On why we gave the ColBERT paradigm the name "late interaction" instead of "multi-vector", a term that emerged later and that has proven to be more intuitive.
**The mechanism is actually not about having multiple vectors at all.** You can see this in four different ways.
🧵1/7
**The mechanism is actually not about having multiple vectors at all.** You can see this in four different ways.
🧵1/7
February 26, 2025 at 7:14 PM
Some quick thoughts: On why we gave the ColBERT paradigm the name "late interaction" instead of "multi-vector", a term that emerged later and that has proven to be more intuitive.
**The mechanism is actually not about having multiple vectors at all.** You can see this in four different ways.
🧵1/7
**The mechanism is actually not about having multiple vectors at all.** You can see this in four different ways.
🧵1/7
Someone needs to write the book “Modern Machine Learning — the math, the myth, the legend”
February 1, 2025 at 5:40 AM
Someone needs to write the book “Modern Machine Learning — the math, the myth, the legend”
a statistician walks into an error bar
surprisingly, not everyone was mean to him
surprisingly, not everyone was mean to him
January 28, 2025 at 8:31 PM
a statistician walks into an error bar
surprisingly, not everyone was mean to him
surprisingly, not everyone was mean to him
What do you call LLMs that exhibit reward hacking for translation?
Agents gone ROUGE.
Agents gone ROUGE.
January 5, 2025 at 3:15 AM
What do you call LLMs that exhibit reward hacking for translation?
Agents gone ROUGE.
Agents gone ROUGE.
Reminder to self:
Much like a good paper always starts with (i) the status of the field at the time of the proposal, (ii) what’s an annoying gap, and (iii) the novel intuition that motivates a new insightful proposal, a good course lecture must also situate every major concept this way.
[1/3]
Much like a good paper always starts with (i) the status of the field at the time of the proposal, (ii) what’s an annoying gap, and (iii) the novel intuition that motivates a new insightful proposal, a good course lecture must also situate every major concept this way.
[1/3]
December 31, 2024 at 9:21 PM
Reminder to self:
Much like a good paper always starts with (i) the status of the field at the time of the proposal, (ii) what’s an annoying gap, and (iii) the novel intuition that motivates a new insightful proposal, a good course lecture must also situate every major concept this way.
[1/3]
Much like a good paper always starts with (i) the status of the field at the time of the proposal, (ii) what’s an annoying gap, and (iii) the novel intuition that motivates a new insightful proposal, a good course lecture must also situate every major concept this way.
[1/3]
When building ColBERT, I sort of assumed it will pave the way for hypernetwork-based, pruning-capable retrieval indexes. Let me explain.
The big insight in ColBERT is that we can encode each document upfront *not* into a vector, but into a rich scoring function, f: query -> float, which ... [1/3]
The big insight in ColBERT is that we can encode each document upfront *not* into a vector, but into a rich scoring function, f: query -> float, which ... [1/3]
December 31, 2024 at 5:29 PM
When building ColBERT, I sort of assumed it will pave the way for hypernetwork-based, pruning-capable retrieval indexes. Let me explain.
The big insight in ColBERT is that we can encode each document upfront *not* into a vector, but into a rich scoring function, f: query -> float, which ... [1/3]
The big insight in ColBERT is that we can encode each document upfront *not* into a vector, but into a rich scoring function, f: query -> float, which ... [1/3]
Reposted by Omar Khattab
Anyone here have experience with working with someone in @aclmeeting.bsky.social to get a confirmation of reviewing/awards/etc for the purpose of US work visa / green card? If so, please can you share their contact with me 🙏
December 29, 2024 at 6:54 PM
Anyone here have experience with working with someone in @aclmeeting.bsky.social to get a confirmation of reviewing/awards/etc for the purpose of US work visa / green card? If so, please can you share their contact with me 🙏
Reposted by Omar Khattab
Strongly agree. Especially when you need to be nimble enough to adopt new or different models.
"prompt engineering" is much more about effective processes for creating good prompts for your data, than it is about specific techniques. which is to a large extent why platforms like dspy work: they kinda force you into a processes, while attempting to automate the techniques.
December 29, 2024 at 5:35 PM
Strongly agree. Especially when you need to be nimble enough to adopt new or different models.