Nikhil Benesch
@benesch.bsky.social
Systems engineer @turbopuffer.bsky.social. Former CTO @materialize.com.
I did! Couldn't pass up the opportunity to work with this team!
March 7, 2025 at 7:26 PM
I did! Couldn't pass up the opportunity to work with this team!
Of course! Will be very excited to give the new DynamoDB-free SlateDB a spin.
December 19, 2024 at 5:03 AM
Of course! Will be very excited to give the new DynamoDB-free SlateDB a spin.
Same! I’ve been using Amethyst for over ten years at this point. When I discovered the project in 2013 I never expected that ianyh would still be lovingly maintaining it a decade later. One of my most loved pieces of software, for sure.
December 11, 2024 at 5:17 AM
Same! I’ve been using Amethyst for over ten years at this point. When I discovered the project in 2013 I never expected that ianyh would still be lovingly maintaining it a decade later. One of my most loved pieces of software, for sure.
tl;dr the 10x TPS claim actually *is* based on a small but novel optimization inside of S3! Table buckets understand Iceberg naming conventions and adjust S3's rate limiting so that you start with budget for 55k/35k GETs/PUTs per sec rather than the general purpose default of 5.5k/3.5k.
December 11, 2024 at 1:59 AM
tl;dr the 10x TPS claim actually *is* based on a small but novel optimization inside of S3! Table buckets understand Iceberg naming conventions and adjust S3's rate limiting so that you start with budget for 55k/35k GETs/PUTs per sec rather than the general purpose default of 5.5k/3.5k.
It just occurred to me that while the 3x query performance claim is based on a comparison to uncomplicated tables, I hadn't actually seen the evidence for the 10x TPS claim.
After some digging I just found the explanation in the re:Invent talk (start at 14:24): youtu.be/1U7yX4HTLCI?...
After some digging I just found the explanation in the re:Invent talk (start at 14:24): youtu.be/1U7yX4HTLCI?...
AWS re:Invent 2024 - [NEW LAUNCH] Store tabular data at scale with Amazon S3 Tables (STG367-NEW)
YouTube video by AWS Events
youtu.be
December 11, 2024 at 1:59 AM
It just occurred to me that while the 3x query performance claim is based on a comparison to uncomplicated tables, I hadn't actually seen the evidence for the 10x TPS claim.
After some digging I just found the explanation in the re:Invent talk (start at 14:24): youtu.be/1U7yX4HTLCI?...
After some digging I just found the explanation in the re:Invent talk (start at 14:24): youtu.be/1U7yX4HTLCI?...
Exciting! I'm so here for it. 👀
December 11, 2024 at 1:50 AM
Exciting! I'm so here for it. 👀
Disclaimer: I'm not involved with DuckDB, ClickHouse, or Iceberg, so take this all with a grain of salt.
December 10, 2024 at 6:27 PM
Disclaimer: I'm not involved with DuckDB, ClickHouse, or Iceberg, so take this all with a grain of salt.
Rust might be feasible! In fact I think DuckDB's Delta extension is based on a Rust library. And ClickHouse is starting to integrate some Rust too: clickhouse.com/blog/more-th...
Sadly Rust's Iceberg library is still relatively immature (e.g. no support for writes: github.com/apache/icebe...).
Sadly Rust's Iceberg library is still relatively immature (e.g. no support for writes: github.com/apache/icebe...).
More Than 2x Faster Hashing in ClickHouse Using Rust
Rust’s rich type system and ownership model guarantee memory-safety and thread-safety. There are a fairly large number of useful libraries written on it, so we considered using them in ClickHouse.
clickhouse.com
December 10, 2024 at 6:27 PM
Rust might be feasible! In fact I think DuckDB's Delta extension is based on a Rust library. And ClickHouse is starting to integrate some Rust too: clickhouse.com/blog/more-th...
Sadly Rust's Iceberg library is still relatively immature (e.g. no support for writes: github.com/apache/icebe...).
Sadly Rust's Iceberg library is still relatively immature (e.g. no support for writes: github.com/apache/icebe...).
Calling Go code from another language is hard enough that' it's rarely done—especially in high performance contexts. (🌶️ alert, but fasterthanli.me/articles/lie... covers this—see "the only good boundary with Go is a network boundary.")
Lies we tell ourselves to keep using Golang
In the two years since I've posted I want off Mr Golang's Wild Ride, it's made the rounds time and time again, on Reddit, on Lobste.rs, on HackerNews, and elsewhere. And every time, it elicits the ...
fasterthanli.me
December 10, 2024 at 6:27 PM
Calling Go code from another language is hard enough that' it's rarely done—especially in high performance contexts. (🌶️ alert, but fasterthanli.me/articles/lie... covers this—see "the only good boundary with Go is a network boundary.")
Nice, looking forward to it!
December 10, 2024 at 3:28 PM
Nice, looking forward to it!
I’m not 100% sure on the chronology but I believe DuckDB originated many of these pragmatic SQL UX improvements: duckdb.org/2022/05/04/f...
Friendlier SQL with DuckDB
DuckDB offers several extensions to the SQL syntax. For a full list of these features, see the Friendly SQL documentation page.
duckdb.org
December 10, 2024 at 2:11 PM
I’m not 100% sure on the chronology but I believe DuckDB originated many of these pragmatic SQL UX improvements: duckdb.org/2022/05/04/f...
Keep an eye on github.com/apache/icebe...! AFAIU the lack of a C++ Iceberg library is what’s been holding back full Iceberg support in DuckDB and ClickHouse.
GitHub - apache/iceberg-cpp: Apache Iceberg C++
Apache Iceberg C++. Contribute to apache/iceberg-cpp development by creating an account on GitHub.
github.com
December 10, 2024 at 2:10 PM
Keep an eye on github.com/apache/icebe...! AFAIU the lack of a C++ Iceberg library is what’s been holding back full Iceberg support in DuckDB and ClickHouse.
@chris.blue you might be interested in this. Something like this fits in very neatly to your views on the commoditization of the PostgreSQL dialect/protocol (materializedview.io/p/databases-...).
Databases Are Commodities. Now What?
How will vendors differentiate when databases are commoditized? I've got three ideas.
materializedview.io
December 10, 2024 at 6:54 AM
@chris.blue you might be interested in this. Something like this fits in very neatly to your views on the commoditization of the PostgreSQL dialect/protocol (materializedview.io/p/databases-...).