(3/3) As a data engineer, I could have spent my time learning one of the shiny tools out there or could have relied on co-pilot to write code. I chose to learn what was happening under the hood in Python, mainly in the interest of building a stronger foundation, which should be a priority.

(2/3) I am no better and I am guilty of this myself, mainly from my own ignorance to writing Python idiomatically. This is why I spent the last 6 months re-learning Python, consuming multiple books, attending advanced lessons on specific topics, and wrote a #data ingestion package using Python.

(4/4) It is crazy to think that this news came out just 2 weeks into the new year. I cannot wait to see what dbt and others have in store for the next 12 months. I am excited to see how the integration between sdf and dbt will be rolled out. Maybe we will have some updates at Coalesce 2025!!!

(3/4) I have always found the incredibly slow compile times with dbt deeply frustrating, which is of no concern to business users. But considering the early adoption of dbt is strongly rooted in the technical community, the acquisition of SDF to improve the developer experience is well timed.

(2/4) I have used dbt quite extensively, and do enjoy its utility when it comes to data transformations and acknowledge that it is not going anywhere. However, I do feel that dbt core have not had any significant upgrade in a while, especially when it comes to improving its developer experience.

(3/3) I am sure this will change as we iterate through the next generation of language models. However, if you are someone that is just starting out in the world of #data, I recommend you to reduce your reliance on code assist and instead spend some time understanding the basics of how SQL works.

(2/2) But if you had to work with complex #SQL statements such as first touch attribution, finding the streak, calculating conversions etc with highly specific business logic and questionable quality of data, you probably will understand the sentiment that LLMs we have today are not powerful enough.

The thing that allows you to link data from the physical world in a format that a machine can understand coherently.

What did they buy before?

SDF was on my list of things to try out. I'll just wait till they integrate it to dbt now I guess :)

(3/3) Laktory also supports managing ETL pipelines, much like how you would do with dbt but with Spark and/or SQL. What I really like about Laktory is its ability to modularize the Databricks assets, which is big win when it comes to long term maintainability of your #data platform.

(2/3) I am not affiliated with Laktory but if you are someone who works with Databricks and want to take a break from having to wrestle with Terraform/Pulumi, check out Laktory. It is an absolute game changer and I was able to go zero to managing multiple workspaces with a couple of yaml files.

This is the approach I default to. Also, I always thought that this was the only way OBT was used. I guess not.

To an extent, I agree. Though I still prefer to use documentation alongside AI assistance to verify some of the output.

I doubt there is one course that covers them all. You can always pick what part of DE you want to learn and drill down on that first. So that would be for things like ingestion, then transformation,, orchestration and so on

Senior data engineer!! That's tricky almost all of the content out there is generally catered towards early stage DEs. But I have heard Zach Wilson's boot camp is pretty good for seniors.

Disclaimer - I have not attended the boot camp, so take my advice with a grain of salt.

(2/2) I am no where near mastering it but I am happy that I was able to use it recently to build out a data platform on #Databricks and it was not as challenging as I imagined. Key takeaway, being comfortable with IaC can dramatically improve the efficiency and reliability of your #data pipelines.