Perhaps using unnest?
select unnest(value, recursive := true) from read_json('~/Data/example.json')

My blogging motivation has declined a lot over the years... then the endless Hugo breaking changes pretty much killed it for good. Do you know how much work it would be to convert a Hugo blog over to Zola?

I met my wife on xanga! ♥️

Chispa has good diffs for PySpark dataframes
github.com/MrPowers/chi...

I had an HDMI KVM but it was still annoying to switch back and forth. Plus I wanted to use the full resolution at 144Hz on my gaming PC. Now I just have a big desk with separate keyboards/mice/monitors.

Yes, I'm working on this right now and talking about how we can potentially "upgrade" some of the dimensions without breaking everything. 🙃

Thanks for the input and I agree... Right now I'm battling a mono-repo used by a big team with limited git knowledge and no tooling. Choosing a tool like dbt/flyway/liquibase could help force some standardization.

Do you ever feel like it is difficult to keep them in sync with what is deployed to the database? Or with many people working in the same repo?

Is keeping the table definition valuable for only certain databases? For example in Databricks you can easily get the definition and there aren't any indexes to store. Compared to SQL Server (or similar) where it is difficult to figure out what was deployed.

You did this only for certain breaking changes right? For example - meaning of the data in a column changed or columns removed. How did you maintain two separate versions of the schema?

In my experience I'm seeing companies using Spark switch from Scala to Python for two reasons: Python has an easier learning curve and Scala devs are much harder to find.