Mistakes help us grow, whether ours or others'. In data, small errors can cause big issues like broken pipelines or high costs. These lessons aren’t just for data engineers, they benefit anyone working with data.
#datasky
#dataBS

Mistakes help us grow, whether ours or others'. In data, small errors can cause big issues like broken pipelines or high costs. These lessons aren’t just for data engineers, they benefit anyone working with data.
#datasky
#dataBS

Data Compression in SQL
In this post, we’ll explore:
- What is Data Compression?
- Benefits of Data Compression in SQL.
- Types of Compression in SQL.
- Comparison of SQL Compression Techniques.
- When to Use Compression and When to Avoid.
#databs #datasky

What is Zero-ETL ? What it isn’t ?
Read more here :
open.substack.com/pub/pipeline...

#dataBS
#datasky

In this post, we will cover:

- What is Data Vault 2.0
- Common Data Vault interview questions
- Bridging the fundamental models to Data Vault
- A Case Study: Converting a Dimensional Model to a Data Vault

#dataBS #datasky

Explore key data modelling concepts for databases and data warehouses, including:
- Normalisation vs. Denormalisation
- 3NF
- Dimensional Modeling
- Star vs. Snowflake Schema comparisons.
#databs
#datasky

11 Storage Formats for Data Engineers
Efficient data starts with the right storage format. Explore 11 formats every data engineer should know to match workloads and scale seamlessly.
Highlights: Row & Columnar, Key-Value, Document, Graph, Time-Series, Hybrid.

#databs
#datasky

Data ingestion with dlt and Dagster: An end-to-end pipeline tutorial:

Curious like us to see what people are sharing with #dataBS and #datasky? Check out this post to learn how to do it using dlt!"
@matthausk.bsky.social
@datateam.bsky.social
@hgeren.bsky.social
@hopefanhe.bsky.social
#dlt

Week 6 of '100 Days of SQL Optimisation':
Focused on DuckDB, leveraging columnar storage, sorted data, temp tables, Parquet, and optimal data types to boost efficiency. See how in-memory execution and smart structures enhance query performance!

@duckdb.org
#dataBS
#datasky
#duckdb

Protect data with encryption, access controls, and monitoring. Safeguard credentials, apply least privilege, store only essential sensitive data, and ensure cloud security with IAM and encryption. Build a culture of security beyond compliance.
@joereis.bsky.social

We are starting a 32-week Data Engineering Interview Guide program, covering everything from fundamentals to advanced topics, with sessions every Saturday.
Do you think we're missing any critical topics? We're curious about your opinions😊
#dataBS
#datasky

As a Data Engineer, understanding the data storage lifecycle and data retention policies is critical for designing efficient, cost-effective, and compliant data systems.
@joereis.bsky.social
#dataBS #datasky

substack.com/@pipeline2in...

In our new post, we've covered 10 of the most popular data pipeline design patterns.

We’d love to hear your thoughts. For more details, please check out the full post created by (@hgeren.bsky.social and @hopefanhe.bsky.social ): open.substack.com/pub/pipeline...

#dataBS #datasky

Discover how dlt simplifies data ingestion.
Learn its origins and real-world use cases. Follow a step-by-step guide to build your first pipeline and join the growing dlt community!
@matthausk.bsky.social
@datateam.bsky.social
@hgeren.bsky.social
@hopefanhe.bsky.social

#dataBS #datasky

Hi, wishing everyone a great Thanksgiving!

Recently we wrote about how SQL queries are executed behind the scenes.

If you are interested, check out our post: open.substack.com/pub/pipeline...

#dataBS #datasky

Just joined and heard #dataBS and #datasky are where the cool kids hang.

Wanted to introduce our blog where we regularly write about Data Engineering concepts, news, and tools.

pipeline2insights.substack.com

Storage is at the heart of Data Engineering.
In this post, we explore the hierarchy of data storage from the ground up, drawing inspiration from Fundamentals of Data Engineering by
@joereis.bsky.social
and Matt Housley, as well as insights from the DE Professionals on Coursera.
#dataBS #datasky