Tristan Handy
@jthandy.bsky.social
dbt
Always trying to make something that means something. Ziplines, sailboats, kids, companies, relationships.
Always trying to make something that means something. Ziplines, sailboats, kids, companies, relationships.
As we work through integration details we’ll share more about how this will work. But if you use dbt today, you’ll be able to use this new tech.
January 14, 2025 at 2:22 PM
As we work through integration details we’ll share more about how this will work. But if you use dbt today, you’ll be able to use this new tech.
While SDF won't be included as part of the Apache 2.0 code base, we plan to make meaningful parts of SDF’s capabilities available to all dbt users—whether you’re using dbt Core or dbt Cloud.
January 14, 2025 at 2:22 PM
While SDF won't be included as part of the Apache 2.0 code base, we plan to make meaningful parts of SDF’s capabilities available to all dbt users—whether you’re using dbt Core or dbt Cloud.
So what does this mean for dbt users? The first goal is to get SDF’s SQL parsing capabilities integrated into dbt.
January 14, 2025 at 2:22 PM
So what does this mean for dbt users? The first goal is to get SDF’s SQL parsing capabilities integrated into dbt.
Local Execution: Instead of having to hit your data platform in development, you can take that logical plan and execute it in a local environment.
January 14, 2025 at 2:22 PM
Local Execution: Instead of having to hit your data platform in development, you can take that logical plan and execute it in a local environment.
Lineage: SDF has both the highest-fidelity and most high-performing SQL parsing on the market. And lineage and metadata is, of course, at the heart of the entire data control plane.
January 14, 2025 at 2:22 PM
Lineage: SDF has both the highest-fidelity and most high-performing SQL parsing on the market. And lineage and metadata is, of course, at the heart of the entire data control plane.
Because SDF understands your SQL, it can detect errors without connecting to the remote database. Troubleshooting all of a sudden becomes far faster, as errors get caught as you are typing, not when you do a dbt run.
January 14, 2025 at 2:22 PM
Because SDF understands your SQL, it can detect errors without connecting to the remote database. Troubleshooting all of a sudden becomes far faster, as errors get caught as you are typing, not when you do a dbt run.
SDF’s ability to understand SQL means that it can power IntelliSense in your IDE of choice. With every keystroke, SDF understands what you are typing and can automatically suggest what comes next, including suggesting table and column names.
January 14, 2025 at 2:22 PM
SDF’s ability to understand SQL means that it can power IntelliSense in your IDE of choice. With every keystroke, SDF understands what you are typing and can automatically suggest what comes next, including suggesting table and column names.
Developer experience: There are many things that will eventually go into this bucket, but here are two great examples
January 14, 2025 at 2:22 PM
Developer experience: There are many things that will eventually go into this bucket, but here are two great examples
SDF parses and compiles dbt projects really, really fast: Because it’s built in Rust, it simply runs faster than Python. As a result, SDF compiles the same dbt project multiple orders of magnitude faster than dbt Core. If you’re working in a large dbt project, this will vv impact your productivity.
January 14, 2025 at 2:22 PM
SDF parses and compiles dbt projects really, really fast: Because it’s built in Rust, it simply runs faster than Python. As a result, SDF compiles the same dbt project multiple orders of magnitude faster than dbt Core. If you’re working in a large dbt project, this will vv impact your productivity.
Benefits for developers: faster, a better developer experience, lineage and local execution
January 14, 2025 at 2:22 PM
Benefits for developers: faster, a better developer experience, lineage and local execution
Integration is easy. SDF has adopted dbt’s syntax, configuration, libraries, and Jinja natively, as part of the SDF runtime. As a result, for most dbt projects there will be no code changes required to take full advantage of SDF’s capabilities!
January 14, 2025 at 2:22 PM
Integration is easy. SDF has adopted dbt’s syntax, configuration, libraries, and Jinja natively, as part of the SDF runtime. As a result, for most dbt projects there will be no code changes required to take full advantage of SDF’s capabilities!
Unlike dbt historically (which has treated SQL as strings), SDF sees objects and types and syntax and semantics. In the same way that Virtual Machines (VMs) emulate physical hardware, SDF emulates the SQL compilers native to the data platforms you use.
January 14, 2025 at 2:22 PM
Unlike dbt historically (which has treated SQL as strings), SDF sees objects and types and syntax and semantics. In the same way that Virtual Machines (VMs) emulate physical hardware, SDF emulates the SQL compilers native to the data platforms you use.
The toolchain is powered by a state-of-the-art development in SQL understanding. SDF represents each SQL dialect (Snowflake, Redshift, BigQuery, etc) as a complete ANTLR grammar with definitions for all datatypes, coercion rules, functions, scoping intricacies and more.
January 14, 2025 at 2:22 PM
The toolchain is powered by a state-of-the-art development in SQL understanding. SDF represents each SQL dialect (Snowflake, Redshift, BigQuery, etc) as a complete ANTLR grammar with definitions for all datatypes, coercion rules, functions, scoping intricacies and more.
It is written in Rust, highly parallelized, and designed for scale.
January 14, 2025 at 2:21 PM
It is written in Rust, highly parallelized, and designed for scale.
What is SDF? SDF is a high performance toolchain for SQL development packaged into one CLI; a multi-dialect SQL compiler, type system, transformation framework, linter, and language server.
January 14, 2025 at 2:21 PM
What is SDF? SDF is a high performance toolchain for SQL development packaged into one CLI; a multi-dialect SQL compiler, type system, transformation framework, linter, and language server.
jealous. i never got this upgrade. i just rely on my 'doesn't need that much sleep' superpower :P
November 25, 2024 at 7:15 PM
jealous. i never got this upgrade. i just rely on my 'doesn't need that much sleep' superpower :P
Yep, agree. Unsurprisingly I see a lot of this as an ecosystem problem and think SWE is ahead because of the persistent compounding effects of OSS over the course of multiple decades. I think data people are constantly forced to make this shit choice when they should be able to have both.
November 22, 2024 at 9:41 PM
Yep, agree. Unsurprisingly I see a lot of this as an ecosystem problem and think SWE is ahead because of the persistent compounding effects of OSS over the course of multiple decades. I think data people are constantly forced to make this shit choice when they should be able to have both.
sorry, legit not meant as a snipe, i found your post provocative.
November 22, 2024 at 1:59 AM
sorry, legit not meant as a snipe, i found your post provocative.
What is next? Do you go into prod also in Duck? or..?
Real question, curious about the workflow you're cooking on.
Real question, curious about the workflow you're cooking on.
November 19, 2024 at 6:02 PM
What is next? Do you go into prod also in Duck? or..?
Real question, curious about the workflow you're cooking on.
Real question, curious about the workflow you're cooking on.
Reposted by Tristan Handy
Don’t know what statistical process control is? This podcast where @jthandy.bsky.social interviews @cedricchin.bsky.social will get you started
overcast.fm/+AAw94Uafve8
overcast.fm/+AAw94Uafve8
Data as an assembly line (w/ Cedric Chin) — The Analytics Engineering Podcast
Cedric Chin runs Commoncog—a publication about accelerating business expertise. He joins Tristan to talk about the analytics development lifecycle, how organizations value (or misvalue) data, and why ...
overcast.fm
November 19, 2024 at 1:04 AM
Don’t know what statistical process control is? This podcast where @jthandy.bsky.social interviews @cedricchin.bsky.social will get you started
overcast.fm/+AAw94Uafve8
overcast.fm/+AAw94Uafve8