Not sure about the #databricks assistant. I had an argument with it today about UNPIVOT. It is convinced that Databricks SQL doesn't support it and suggests I write UNION ALL for each column. If it has to do one thing, it's read its own bloody documentation.

is this any good?

WITH data AS (
SELECT json_extract(json, '$.value') AS v
FROM my_data),
kv AS (
SELECT
key,
json_extract_string(value, '$.string') AS val
FROM data, json_each(v)
WHERE json_extract_string(value, '$.string') IS NOT NULL
)
PIVOT kv ON key USING (MAX(val))

i have a similar stack @dagster.io @getdbt.com dlt and databricks in azure. Dagtster and dbt work together like a dream. love @duckdb.org for local dev and poc wrangling. might try motherduck at some point especially when ducklake has matured. using k8s and grafana for alerts

this is the best tech tip i've seen for many years☺️

FYI i've posted a link to my dbt cheat sheet which might help
bsky.app/profile/stre...

feels like the perfect sized project to get you going. dbt has so many amazing features its well worth the effort and will save you so much time if you have anything more complex

Nice work. I love @duckdb.org is easy and fast to do analytical stuff. if you've not already tried it have a go with @getdbt.com its also an OSS and could make your ropey cron job 😁 and 14 sql statements into an elegant documented pipeline.

I hope they haven't just enrolled into squid game series 3

And lastly if I host my parquet files in azure its way slower to run the pipe than if the files are local.
All that said its way faster and cheaper to do my dev than having to start and stop Databricks. Loving the duck

Only irritations are I like to use dbeaver to check my data and I have to keep disconnecting to run dbt. On occasion even need to quit dbeaver to release the file lock. I've also see dbt get locked out of the db by itself which is very odd as everything is single thread.

Nice. I'm also doing this with dbt but with duckdb and databricks. Some minor macros to help with some syntax diffs. However the dbt sdf news means that might be a thing of the past.
I've also got most of it running in dagster