It’s interesting the tradeoffs if the main goal is no operating cost and decent startup time. Definitely painful to develop on regularly but for a one and done this makes a lot of sense. I wonder if Rust compile times will come down further one day.

I didn’t quite make it in time for Hive filtering lazy list to speed up filtering Hive folder with many partitions, but will pick up again before next release w/ luck 🙇‍♂️ github.com/duckdb/duckd...

Yeah, I don’t think MS is interested in 3rd party devs so much.

Unfortunately yes, I was already going to get one for something else and this put me over the edge. Maybe I'll also build a gaming rig one day in the distant future haha.

Just the 1 day of data above is ~125GiB compressed, ~585GiB uncompressed. One month is about 3.75TiB compressed, or 17.5TiB. It makes sense this dataset is so popular for testing and analysis, wow.

Uploaded a simplified query. Had to delete and repost since no edit button on the snippets site, sorry for the spam haha.

Haha already posted in case someone benefits there too.

Automator using simple shell script to call sqlfluff. Added keyboard shortcut for the service too. Easier than making browser extensions for each browser, although unfortunately not cross-platform.

The pieces were already there, but progress is not always linear. Majority of cases now handled by [Mother]Duck[DB|Lake], Spark for extreme cases. Single-node compute 10 years from now is going to be mindblowing, but already exciting what we can do today.