Ben Trent
@benwtrent.bsky.social
Doer of things | Builder of things | software engineer
@elastic
@elastic
The @hf.co community is awesome. Real work that moves everyone forward: huggingface.co/blog/rteb
Introducing RTEB: A New Standard for Retrieval Evaluation
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
October 1, 2025 at 4:22 PM
The @hf.co community is awesome. Real work that moves everyone forward: huggingface.co/blog/rteb
Apache Lucene 10.3.0 is released! 40% faster lexical search is absolutely crazy for a project that has been doing lexical search for a quarter of a century lucene.apache.org/core/corenew...
Lucene™ Core News
Apache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for...
lucene.apache.org
September 19, 2025 at 8:20 PM
Apache Lucene 10.3.0 is released! 40% faster lexical search is absolutely crazy for a project that has been doing lexical search for a quarter of a century lucene.apache.org/core/corenew...
Storing floating point values as a big 'ole JSON blob is silly, so we stopped doing that. Great stuff from Jim on making vector search in Elasticsearch substantially cheaper! www.elastic.co/search-labs/...
Elasticsearch vector search: Excluding vectors from source - Elasticsearch Labs
Elasticsearch now excludes vectors from source by default, saving space and improving performance while keeping vectors accessible when needed.
www.elastic.co
August 27, 2025 at 1:42 PM
Storing floating point values as a big 'ole JSON blob is silly, so we stopped doing that. Great stuff from Jim on making vector search in Elasticsearch substantially cheaper! www.elastic.co/search-labs/...
Reposted by Ben Trent
Next in the series of building a search engine from scratch - we focus on hybrid retrieval with @benwtrent.bsky.socialof Elastic.
How do you add filtering to a vector search index?
I'll code. He'll yell at me.
maven.com/p/430592/hyb...
How do you add filtering to a vector search index?
I'll code. He'll yell at me.
maven.com/p/430592/hyb...
Hybrid search live coded from scratch
RAG systems all use vector databases. HNSW (Hierarchical Navigable Small Worlds) is the most common algorithm. If you want to build RAG, you should appreciate how this algorithm works (Missed previous...
maven.com
May 21, 2025 at 1:15 PM
Next in the series of building a search engine from scratch - we focus on hybrid retrieval with @benwtrent.bsky.socialof Elastic.
How do you add filtering to a vector search index?
I'll code. He'll yell at me.
maven.com/p/430592/hyb...
How do you add filtering to a vector search index?
I'll code. He'll yell at me.
maven.com/p/430592/hyb...
Reposted by Ben Trent
It's time to redo benchmarks! #Lucene 10.2 was just released, with
- huge speedups to non-scoring boolean queries, range queries and filtered vector search,
- better merging defaults for faster search,
- much faster merging of vectors
And more...
lucene.apache.org/core/corenew...
- huge speedups to non-scoring boolean queries, range queries and filtered vector search,
- better merging defaults for faster search,
- much faster merging of vectors
And more...
lucene.apache.org/core/corenew...
Lucene™ Core News
Apache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for...
lucene.apache.org
April 12, 2025 at 6:27 AM
It's time to redo benchmarks! #Lucene 10.2 was just released, with
- huge speedups to non-scoring boolean queries, range queries and filtered vector search,
- better merging defaults for faster search,
- much faster merging of vectors
And more...
lucene.apache.org/core/corenew...
- huge speedups to non-scoring boolean queries, range queries and filtered vector search,
- better merging defaults for faster search,
- much faster merging of vectors
And more...
lucene.apache.org/core/corenew...
Filtered vector search is crazy important. So we made HNSW filtered search in Apache Lucene better. At similar recall, it can be 3-5x faster!
February 28, 2025 at 3:39 PM
Filtered vector search is crazy important. So we made HNSW filtered search in Apache Lucene better. At similar recall, it can be 3-5x faster!
Reposted by Ben Trent
"elasticsearch: 15 years of indexing it all, finding what matters": www.elastic.co/search-labs/...
we turned it into a proper blog post with shay :)
we turned it into a proper blog post with shay :)
Elasticsearch history: 15 years of indexing and searching - Elasticsearch Labs
Elasticsearch just turned 15-years-old! Take a look back at the last 15 years of indexing and searching, and turn to the next 15 years of relevance.
www.elastic.co
February 13, 2025 at 11:54 PM
"elasticsearch: 15 years of indexing it all, finding what matters": www.elastic.co/search-labs/...
we turned it into a proper blog post with shay :)
we turned it into a proper blog post with shay :)
I really enjoyed this talk by @elasticmark.bsky.social. He is back at finding crazy & interesting ways to explore data (I guess he never stopped). Clustering with binary vectors & vector search with Elasticsearch www.youtube.com/watch?v=sJU_...
Binary Vectors & Fuzzy Facets: Clustering Results in a Browser Using Binary Vectors
YouTube video by Official Elastic Community
www.youtube.com
February 13, 2025 at 9:57 PM
I really enjoyed this talk by @elasticmark.bsky.social. He is back at finding crazy & interesting ways to explore data (I guess he never stopped). Clustering with binary vectors & vector search with Elasticsearch www.youtube.com/watch?v=sJU_...
It's wonderful to see practical & important programming work. Debugging concurrent programs is incredibly difficult, here is a bug found in Apache Lucene by the CMU Pasta Lab using their new Fray testing framework www.elastic.co/search-labs/...
Concurrency bugs in Lucene: How to fix optimistic concurrency failures - Elasticsearch Labs
Thanks to Fray, a deterministic concurrency testing framework from CMU’s PASTA Lab, we tracked down a tricky Lucene bug and squashed it
www.elastic.co
February 7, 2025 at 3:59 PM
It's wonderful to see practical & important programming work. Debugging concurrent programs is incredibly difficult, here is a bug found in Apache Lucene by the CMU Pasta Lab using their new Fray testing framework www.elastic.co/search-labs/...
The number of improvements in Lucene here are crazy. Pretty much every count and boolean query gets a nice boost and some of the count improvements are hilarious 🚀🚀🚀.
January 15, 2025 at 6:28 PM
The number of improvements in Lucene here are crazy. Pretty much every count and boolean query gets a nice boost and some of the count improvements are hilarious 🚀🚀🚀.
It's so cool to see #Apache #Lucene going strong after about a quarter of a century 🤯. 2025 is gonna be a fun year for Lucene. www.elastic.co/search-labs/...
Lucene Wrapped 2024 - Elasticsearch Labs
2024 has been another major year for Apache Lucene. In this blog, we’ll explore the key highlights.
www.elastic.co
January 10, 2025 at 1:32 PM
It's so cool to see #Apache #Lucene going strong after about a quarter of a century 🤯. 2025 is gonna be a fun year for Lucene. www.elastic.co/search-labs/...
Early termination for vector search can be more than just "gathering K candidates" my colleague Tommaso gives a small overview of basic early termination strategies for vector index search. www.elastic.co/search-labs/...
Early termination in HNSW for faster approximate KNN search - Elasticsearch Labs
Learn how HNSW can be made faster for KNN search, using smart early termination strategies.
www.elastic.co
January 7, 2025 at 3:19 PM
Early termination for vector search can be more than just "gathering K candidates" my colleague Tommaso gives a small overview of basic early termination strategies for vector index search. www.elastic.co/search-labs/...
My team wrote a new backing algorithm for our BBQ indices, called Optimized Scalar Quantization. Here is a high level overview of its implementation in Elasticsearch (and soon Apache Lucene). www.elastic.co/search-labs/... for the math nerds, skip to Tom's blog: www.elastic.co/search-labs/...
Optimized Scalar Quantization: Even Better Binary Quantization - Elasticsearch Labs
Here we explain optimized scalar quantization in Elasticsearch and how we used it to improve Better Binary Quantization (BBQ).
www.elastic.co
January 6, 2025 at 6:13 PM
My team wrote a new backing algorithm for our BBQ indices, called Optimized Scalar Quantization. Here is a high level overview of its implementation in Elasticsearch (and soon Apache Lucene). www.elastic.co/search-labs/... for the math nerds, skip to Tom's blog: www.elastic.co/search-labs/...
Reposted by Ben Trent
Lucene has been evaluating disjunctive queries by loading (windows of) postings into a bit set and or-ing these bit sets for 20+ years. It started using the same approach for conjunctive queries a few days ago. benchmarks.mikemccandless.com/CountAndHigh... (annotation HS)
Lucene CountAndHighHigh queries/sec
benchmarks.mikemccandless.com
December 21, 2024 at 4:37 PM
Lucene has been evaluating disjunctive queries by loading (windows of) postings into a bit set and or-ing these bit sets for 20+ years. It started using the same approach for conjunctive queries a few days ago. benchmarks.mikemccandless.com/CountAndHigh... (annotation HS)
Something a little different from my typical blogs. This line of code in Apache Lucene took me 3 days to write. For fixing bugs, it's about the journey, not necessarily the destination. www.elastic.co/search-labs/... (the cover art was provided by one of my kids :))
Lucene bug adventures: Fixing a corrupted index exception - Elasticsearch Labs
Sometimes, a single line of code takes days to write. Here, we get a glimpse of an engineer's pain and debugging over multiple days to fix a potential Apache Lucene index corruption.
www.elastic.co
December 27, 2024 at 5:16 PM
Something a little different from my typical blogs. This line of code in Apache Lucene took me 3 days to write. For fixing bugs, it's about the journey, not necessarily the destination. www.elastic.co/search-labs/... (the cover art was provided by one of my kids :))
Our Better Binary Quantization (BBQ) index in Elasticsearch has a new backing algorithm. Better(er) recall & query speed for vector search. Its a natural evolution of our scalar quantization. Shipping soon. It's pretty neat www.elastic.co/search-labs/...
Understanding optimized scalar quantization - Elasticsearch Labs
In this post we explain a new form of scalar quantization we've developed at Elastic that achieves state-of-the-art accuracy for binary quantization
www.elastic.co
December 20, 2024 at 4:14 PM
Our Better Binary Quantization (BBQ) index in Elasticsearch has a new backing algorithm. Better(er) recall & query speed for vector search. Its a natural evolution of our scalar quantization. Shipping soon. It's pretty neat www.elastic.co/search-labs/...
Elasticsearch just got more powerful. Now, semantic, hybrid, and vector retrieval with custom rules for pinning and bubbling results to the top! Now you have multi-phased, hybrid retrieval in combination with business rules :D www.elastic.co/search-labs/...
Ensuring business rules work seamlessly with semantic search - Elasticsearch Labs
Harness the power of query rules combined with semantic search and rerankers.
www.elastic.co
December 19, 2024 at 3:36 PM
Elasticsearch just got more powerful. Now, semantic, hybrid, and vector retrieval with custom rules for pinning and bubbling results to the top! Now you have multi-phased, hybrid retrieval in combination with business rules :D www.elastic.co/search-labs/...
It was so much fun talking #Elasticsearch with Steve Mayzak on “You Know, For Search”. I could nerd out for hours, but we kept it down to just 1 hour (maybe even that is too long....). Give it a listen, if nothing else, for Steve's dulcet tones: open.spotify.com/episode/7HLH...
Quantization: The Important Bits
You know, for search, an Elastic podcast · Episode
open.spotify.com
December 11, 2024 at 1:07 PM
It was so much fun talking #Elasticsearch with Steve Mayzak on “You Know, For Search”. I could nerd out for hours, but we kept it down to just 1 hour (maybe even that is too long....). Give it a listen, if nothing else, for Steve's dulcet tones: open.spotify.com/episode/7HLH...
Be prepared to learn more about semantic rerankers than you ever thought you needed to know. Another awesome analysis from my colleagues at Elasticsearch www.elastic.co/search-labs/...
Exploring depth in a 'retrieve-and-rerank' pipeline - Elasticsearch Labs
Select an optimal re-ranking depth for your model and dataset.
www.elastic.co
December 5, 2024 at 4:49 PM
Be prepared to learn more about semantic rerankers than you ever thought you needed to know. Another awesome analysis from my colleagues at Elasticsearch www.elastic.co/search-labs/...
More magic from chef Chris Hegarty. How better binary quantization vector ops are accelerated with Java SIMD in Elasticsearch vector search www.elastic.co/search-labs/...
Smokin' fast BBQ with hardware accelerated SIMD instructions - Search Labs
How we optimized vector comparisons in BBQ with hardware accelerated SIMD (Single Instruction Multiple Data) instructions.
www.elastic.co
December 4, 2024 at 7:16 PM
More magic from chef Chris Hegarty. How better binary quantization vector ops are accelerated with Java SIMD in Elasticsearch vector search www.elastic.co/search-labs/...
I cannot adequately express how proud I am of the #Elasticsearch team for delivering this. It is a humungous engineering achievement and the results of (metaphorical) blood, sweat, and (maybe real ;) ) tears. go.es.io/3CVo82X
Do less with serverless: Elastic Cloud Serverless — Now GA
Elastic Cloud Serverless is the easiest way to start and scale your capabilities in search, observability and security. Built on a reimagined Elasticsearch architecture, it ensures low-latency queryin...
go.es.io
December 2, 2024 at 3:08 PM
I cannot adequately express how proud I am of the #Elasticsearch team for delivering this. It is a humungous engineering achievement and the results of (metaphorical) blood, sweat, and (maybe real ;) ) tears. go.es.io/3CVo82X
We have seen this idea played out nicely with Tantivy and Apache Lucene. Benchmarking between each other and lovingly borrowing ideas between the projects.
Open source can be both collaborative and competitive. When a new project shows up, it competes with existing solutions, but that doesn't prohibit collaboration, even between competing projects and ideas.
November 26, 2024 at 9:32 PM
We have seen this idea played out nicely with Tantivy and Apache Lucene. Benchmarking between each other and lovingly borrowing ideas between the projects.
I work with some pretty smart folks. We are shipping a new cross-encoder reranking models. Some pretty great numbers at a moderately sized 183M params. www.elastic.co/search-labs/...
Introducing Elastic Rerank: Elastic's new semantic re-ranker model - Search Labs
Learn about how Elastic's new re-ranker model was trained and how it performs.
www.elastic.co
November 25, 2024 at 6:16 PM
I work with some pretty smart folks. We are shipping a new cross-encoder reranking models. Some pretty great numbers at a moderately sized 183M params. www.elastic.co/search-labs/...
Honestly, Grant Sanderson (3blue1brown) is one of (if not the best) mathematics communicator of my generation. If you want an overview of how LLMs work at a deep & intuitive level, I whole heartedly recommend this lecture: youtu.be/KJtZARuO3JY?...
Visualizing transformers and attention | Talk for TNG Big Tech Day '24
YouTube video by Grant Sanderson
youtu.be
November 22, 2024 at 2:47 PM
Honestly, Grant Sanderson (3blue1brown) is one of (if not the best) mathematics communicator of my generation. If you want an overview of how LLMs work at a deep & intuitive level, I whole heartedly recommend this lecture: youtu.be/KJtZARuO3JY?...