Omar Khattab
@lateinteraction.bsky.social
Incoming asst professor at MIT EECS, Fall 2025. Research scientist at Databricks. CS PhD @StanfordNLP.bsky.social. Author of ColBERT.ai & DSPy.ai.
Btw there’s no trouble in storage at all either.
ColBERT vectors are often 10 bytes each. Ten bytes. That’s like 4 numbers.
It’s not “many vectors work better than one vector”. It’s “set similarity works better than dot product”.
Even with the same storage cost.
ColBERT vectors are often 10 bytes each. Ten bytes. That’s like 4 numbers.
It’s not “many vectors work better than one vector”. It’s “set similarity works better than dot product”.
Even with the same storage cost.
September 28, 2025 at 2:20 AM
Btw there’s no trouble in storage at all either.
ColBERT vectors are often 10 bytes each. Ten bytes. That’s like 4 numbers.
It’s not “many vectors work better than one vector”. It’s “set similarity works better than dot product”.
Even with the same storage cost.
ColBERT vectors are often 10 bytes each. Ten bytes. That’s like 4 numbers.
It’s not “many vectors work better than one vector”. It’s “set similarity works better than dot product”.
Even with the same storage cost.
History - DSPy
The framework for programming—rather than prompting—language models.
dspy.ai
April 6, 2025 at 9:54 PM
This was built by a long-time DSPy community member!
March 4, 2025 at 12:34 AM
This was built by a long-time DSPy community member!
It doesn't help that the we in ML often only design abstractions leak all kinds of implementation details. Folks often define ML itself in terms of techniques, not problems!
But it's prematurely abstracting that leads to the bitterness of wasted effort, and not "modularity doesn't work for AI". 2/2
But it's prematurely abstracting that leads to the bitterness of wasted effort, and not "modularity doesn't work for AI". 2/2
February 26, 2025 at 10:00 PM
It doesn't help that the we in ML often only design abstractions leak all kinds of implementation details. Folks often define ML itself in terms of techniques, not problems!
But it's prematurely abstracting that leads to the bitterness of wasted effort, and not "modularity doesn't work for AI". 2/2
But it's prematurely abstracting that leads to the bitterness of wasted effort, and not "modularity doesn't work for AI". 2/2
4) By default, IR methods that use "multiple vectors" (e.g., cross-encoders) are unscalable. It seems like a necessary tradeoff, but the fascinating thing in late interaction is that it's easy to implement in asymptotically sub-linear ways, thanks to pruning.
Hope this was useful!
Hope this was useful!
February 26, 2025 at 7:14 PM
4) By default, IR methods that use "multiple vectors" (e.g., cross-encoders) are unscalable. It seems like a necessary tradeoff, but the fascinating thing in late interaction is that it's easy to implement in asymptotically sub-linear ways, thanks to pruning.
Hope this was useful!
Hope this was useful!
3) "Multi-vector" makes it sound like these approaches win because they store "more stuff".
But that's not true: if you look at how aggressive ColBERTv2 representations are compressed, it's often ~20 bytes per vector (like 5 floats), which can be smaller than popular uncompressed single vectors!
But that's not true: if you look at how aggressive ColBERTv2 representations are compressed, it's often ~20 bytes per vector (like 5 floats), which can be smaller than popular uncompressed single vectors!
February 26, 2025 at 7:14 PM
3) "Multi-vector" makes it sound like these approaches win because they store "more stuff".
But that's not true: if you look at how aggressive ColBERTv2 representations are compressed, it's often ~20 bytes per vector (like 5 floats), which can be smaller than popular uncompressed single vectors!
But that's not true: if you look at how aggressive ColBERTv2 representations are compressed, it's often ~20 bytes per vector (like 5 floats), which can be smaller than popular uncompressed single vectors!
For dot products, every time you "fix" one query--document pair, you likely break so many other pairs by moving the query and/or document representations.
For ColBERT, you typically *fix* more than you break because you're moving *tokens* in a much smaller (and far more composable!) space.
For ColBERT, you typically *fix* more than you break because you're moving *tokens* in a much smaller (and far more composable!) space.
February 26, 2025 at 7:14 PM
For dot products, every time you "fix" one query--document pair, you likely break so many other pairs by moving the query and/or document representations.
For ColBERT, you typically *fix* more than you break because you're moving *tokens* in a much smaller (and far more composable!) space.
For ColBERT, you typically *fix* more than you break because you're moving *tokens* in a much smaller (and far more composable!) space.
The problem isn't the vector representation, it's the **learnability of the scoring function**.
A dot product is just very hard to learn. An intuition I learned from Menon et al (2021) is that:
A dot product is just very hard to learn. An intuition I learned from Menon et al (2021) is that:
February 26, 2025 at 7:14 PM
The problem isn't the vector representation, it's the **learnability of the scoring function**.
A dot product is just very hard to learn. An intuition I learned from Menon et al (2021) is that:
A dot product is just very hard to learn. An intuition I learned from Menon et al (2021) is that: