Ananth Packkildurai
@ananthdurai.bsky.social
Editor Data Engineering Weekly; subscribe www.dataengineeringweekly.com. In Prgress, LakeByte
The open source companies built their success on top of open-source platforms, benefited from community contributions and adoption, but now must abandon open-source principles to survive commercially.
November 10, 2025 at 2:47 AM
The open source companies built their success on top of open-source platforms, benefited from community contributions and adoption, but now must abandon open-source principles to survive commercially.
🚀 The 244th edition of Data Engineering Weekly dives into:
AI agents as execution engines, LLM inference economics, databases for AI, personalization, and product evidence.
Read more 👉 www.dataengineeringw...
#DataEngineering #AI #LLMs
AI agents as execution engines, LLM inference economics, databases for AI, personalization, and product evidence.
Read more 👉 www.dataengineeringw...
#DataEngineering #AI #LLMs
Data Engineering Weekly #244
The Weekly Data Engineering Newsletter
www.dataengineeringweekly.com
November 3, 2025 at 9:29 AM
🚀 The 244th edition of Data Engineering Weekly dives into:
AI agents as execution engines, LLM inference economics, databases for AI, personalization, and product evidence.
Read more 👉 www.dataengineeringw...
#DataEngineering #AI #LLMs
AI agents as execution engines, LLM inference economics, databases for AI, personalization, and product evidence.
Read more 👉 www.dataengineeringw...
#DataEngineering #AI #LLMs
Cricket has been India’s greatest force in overcoming centuries of colonial suppression. Today’s Women’s World Cup win echoes the spirit of 1983 — a triumph that will inspire generations to come. 🇮🇳🏆
November 3, 2025 at 12:40 AM
Cricket has been India’s greatest force in overcoming centuries of colonial suppression. Today’s Women’s World Cup win echoes the spirit of 1983 — a triumph that will inspire generations to come. 🇮🇳🏆
This is the most personal essay that I have written in Data Engineering Weekly. I shared a few key moments in my life and how fortunate I was to meet mentors along my professional journey, which shaped my career.
Thinking Like a Data Engineer
A Journey Beyond Code — Toward Systems, Curiosity, and Confidence
www.dataengineeringweekly.com
October 23, 2025 at 12:25 AM
This is the most personal essay that I have written in Data Engineering Weekly. I shared a few key moments in my life and how fortunate I was to meet mentors along my professional journey, which shaped my career.
🚀 Data Vault vs. Dimensional Modeling vs. Medallion Architecture — When viewed through a modern enterprise data lens, these techniques interlock.
I break down how in Part 2 of my “Revisiting the Medallion Architecture” series.
I break down how in Part 2 of my “Revisiting the Medallion Architecture” series.
Revisiting Medallion Architecture: Data Vault in Silver, Dimensional Modeling in Gold
How to Balance Flexibility and Performance in a Modern Data Platform
www.dataengineeringweekly.com
October 17, 2025 at 2:54 PM
🚀 Data Vault vs. Dimensional Modeling vs. Medallion Architecture — When viewed through a modern enterprise data lens, these techniques interlock.
I break down how in Part 2 of my “Revisiting the Medallion Architecture” series.
I break down how in Part 2 of my “Revisiting the Medallion Architecture” series.
Fivetran and dbt form a strong foundation for modern data infrastructure, known for bringing simplicity to complex engineering workflows. That said, calling it “open” data infrastructure feels like a stretch.
October 17, 2025 at 12:02 PM
Fivetran and dbt form a strong foundation for modern data infrastructure, known for bringing simplicity to complex engineering workflows. That said, calling it “open” data infrastructure feels like a stretch.
Should we update the definition of an "Analytical Engineer"?
October 13, 2025 at 5:53 PM
Should we update the definition of an "Analytical Engineer"?
As a data engineer, you can't treat zero-party (consent) and third-party (inferred) data the same way. This distinction is critical for building systems that are scalable, private, and trustworthy.
Here’s my guide:
Here’s my guide:
Engineering Growth: The Data Layers Powering Modern GTM
Building privacy-preserving pipelines that unify zero-, first-, second-, third-, and fourth-party data into a coherent GTM ecosystem.
www.dataengineeringweekly.com
October 9, 2025 at 12:35 AM
As a data engineer, you can't treat zero-party (consent) and third-party (inferred) data the same way. This distinction is critical for building systems that are scalable, private, and trustworthy.
Here’s my guide:
Here’s my guide:
Airbnb: Real-Time Key-Value Store
Airbnb’s next-gen key-value store supports real-time ingestion and bulk uploads with sub-second latency, powering feature stores and fraud detection.
Read the full story here: www.dataengineeringw...
Airbnb’s next-gen key-value store supports real-time ingestion and bulk uploads with sub-second latency, powering feature stores and fraud detection.
Read the full story here: www.dataengineeringw...
October 2, 2025 at 1:00 PM
Airbnb: Real-Time Key-Value Store
Airbnb’s next-gen key-value store supports real-time ingestion and bulk uploads with sub-second latency, powering feature stores and fraud detection.
Read the full story here: www.dataengineeringw...
Airbnb’s next-gen key-value store supports real-time ingestion and bulk uploads with sub-second latency, powering feature stores and fraud detection.
Read the full story here: www.dataengineeringw...
Grab: Partner Gateway Metrics at Sub-Second Speed
Real-time partner analytics at scale is tough. Grab uses Apache Pinot, Kafka–Flink ingestion, partitioning, and Star-tree indexing to cut query latency to <300 ms, enabling efficient API monitoring and fast issue resolution.
Real-time partner analytics at scale is tough. Grab uses Apache Pinot, Kafka–Flink ingestion, partitioning, and Star-tree indexing to cut query latency to <300 ms, enabling efficient API monitoring and fast issue resolution.
October 1, 2025 at 1:00 PM
Grab: Partner Gateway Metrics at Sub-Second Speed
Real-time partner analytics at scale is tough. Grab uses Apache Pinot, Kafka–Flink ingestion, partitioning, and Star-tree indexing to cut query latency to <300 ms, enabling efficient API monitoring and fast issue resolution.
Real-time partner analytics at scale is tough. Grab uses Apache Pinot, Kafka–Flink ingestion, partitioning, and Star-tree indexing to cut query latency to <300 ms, enabling efficient API monitoring and fast issue resolution.
Netflix Muse: Scaling Analytics at Trillion-Row Scale
Netflix evolved its Muse architecture to handle huge datasets efficiently: HyperLogLog sketches, Hollow in-memory feeds, and Druid optimizations cut query latency by ~50% and reduced concurrency load.
Netflix evolved its Muse architecture to handle huge datasets efficiently: HyperLogLog sketches, Hollow in-memory feeds, and Druid optimizations cut query latency by ~50% and reduced concurrency load.
September 30, 2025 at 12:33 PM
Netflix Muse: Scaling Analytics at Trillion-Row Scale
Netflix evolved its Muse architecture to handle huge datasets efficiently: HyperLogLog sketches, Hollow in-memory feeds, and Druid optimizations cut query latency by ~50% and reduced concurrency load.
Netflix evolved its Muse architecture to handle huge datasets efficiently: HyperLogLog sketches, Hollow in-memory feeds, and Druid optimizations cut query latency by ~50% and reduced concurrency load.
⚡ Latency Every Data Streaming Engineer Should Know
“Real-time” has limits—disk, network, and replication delays add up. StreamNative explains latency tiers, common costs, and tuning levers like batching & async processing.
💡 Must-read for data streaming engineers!
“Real-time” has limits—disk, network, and replication delays add up. StreamNative explains latency tiers, common costs, and tuning levers like batching & async processing.
💡 Must-read for data streaming engineers!
September 29, 2025 at 12:33 PM
⚡ Latency Every Data Streaming Engineer Should Know
“Real-time” has limits—disk, network, and replication delays add up. StreamNative explains latency tiers, common costs, and tuning levers like batching & async processing.
💡 Must-read for data streaming engineers!
“Real-time” has limits—disk, network, and replication delays add up. StreamNative explains latency tiers, common costs, and tuning levers like batching & async processing.
💡 Must-read for data streaming engineers!
Reposted by Ananth Packkildurai
I enjoyed this post by @ananthdurai.bsky.social. Does a great job tying a bunch of recent papers and concepts together.
What “Supporting Our AI Overlords” and “Semantic Spacetime” Tell Us About the Future of Data Infrastructure
Connecting agent-first and universal semantic grammar to reimagine data infrastructure beyond the relational model.
www.dataengineeringweekly.com
September 27, 2025 at 5:46 PM
I enjoyed this post by @ananthdurai.bsky.social. Does a great job tying a bunch of recent papers and concepts together.
MCP (Model Context Protocol) promises a new way for LLMs to use tools.
Chris Riccomini argues it mostly reinvents OpenAPI, gRPC & CLIs.
Resources = docs
Tools = RPC
Prompts = configs
So… could MCP have just been a JSON file?
💡 More insights: www.dataengineeringw...
Chris Riccomini argues it mostly reinvents OpenAPI, gRPC & CLIs.
Resources = docs
Tools = RPC
Prompts = configs
So… could MCP have just been a JSON file?
💡 More insights: www.dataengineeringw...
September 27, 2025 at 1:00 PM
MCP (Model Context Protocol) promises a new way for LLMs to use tools.
Chris Riccomini argues it mostly reinvents OpenAPI, gRPC & CLIs.
Resources = docs
Tools = RPC
Prompts = configs
So… could MCP have just been a JSON file?
💡 More insights: www.dataengineeringw...
Chris Riccomini argues it mostly reinvents OpenAPI, gRPC & CLIs.
Resources = docs
Tools = RPC
Prompts = configs
So… could MCP have just been a JSON file?
💡 More insights: www.dataengineeringw...
How Tables Got Smarter: Iceberg → DuckLake. From static snapshots to stream-native updates and catalog-first metadata, tables are evolving fast. Choose by intent, not hype.
Subscribe → www.dataengineeringw...
Full story → medium.com/fresha-da...
Subscribe → www.dataengineeringw...
Full story → medium.com/fresha-da...
September 26, 2025 at 12:33 PM
How Tables Got Smarter: Iceberg → DuckLake. From static snapshots to stream-native updates and catalog-first metadata, tables are evolving fast. Choose by intent, not hype.
Subscribe → www.dataengineeringw...
Full story → medium.com/fresha-da...
Subscribe → www.dataengineeringw...
Full story → medium.com/fresha-da...
I wrote my thoughts on Supporting Our AI Overlords.
What “Supporting Our AI Overlords” and “Semantic Spacetime” Tell Us About the Future of Data Infrastructure
Connecting agent-first and universal semantic grammar to reimagine data infrastructure beyond the relational model.
www.dataengineeringweekly.com
September 25, 2025 at 1:15 PM
I wrote my thoughts on Supporting Our AI Overlords.
How Tables Grew a Brain: Iceberg → DuckLake
Snapshots → incremental → stream-native → catalog-first.
Metadata is the bottleneck.
More insights → www.dataengineeringw...
Full story → medium.com/fresha-da...
Snapshots → incremental → stream-native → catalog-first.
Metadata is the bottleneck.
More insights → www.dataengineeringw...
Full story → medium.com/fresha-da...
September 25, 2025 at 12:33 PM
How Tables Grew a Brain: Iceberg → DuckLake
Snapshots → incremental → stream-native → catalog-first.
Metadata is the bottleneck.
More insights → www.dataengineeringw...
Full story → medium.com/fresha-da...
Snapshots → incremental → stream-native → catalog-first.
Metadata is the bottleneck.
More insights → www.dataengineeringw...
Full story → medium.com/fresha-da...
BlaBlaCar scales like a pro!
dbt Core → Transform like a champ
Airflow → Orchestrate effortlessly
CI/CD → Deploy instantly
Dev Containers → Standardized dev
📖 Full story →medium.com/blablacar...
💡 More insights → Subscribe to DEW
#DataEngineering #dbt #Airflow #CICD #DevContainers
dbt Core → Transform like a champ
Airflow → Orchestrate effortlessly
CI/CD → Deploy instantly
Dev Containers → Standardized dev
📖 Full story →medium.com/blablacar...
💡 More insights → Subscribe to DEW
#DataEngineering #dbt #Airflow #CICD #DevContainers
September 24, 2025 at 12:33 PM
BlaBlaCar scales like a pro!
dbt Core → Transform like a champ
Airflow → Orchestrate effortlessly
CI/CD → Deploy instantly
Dev Containers → Standardized dev
📖 Full story →medium.com/blablacar...
💡 More insights → Subscribe to DEW
#DataEngineering #dbt #Airflow #CICD #DevContainers
dbt Core → Transform like a champ
Airflow → Orchestrate effortlessly
CI/CD → Deploy instantly
Dev Containers → Standardized dev
📖 Full story →medium.com/blablacar...
💡 More insights → Subscribe to DEW
#DataEngineering #dbt #Airflow #CICD #DevContainers
🚀 AI adoption is booming—but most data isn’t ready!
AI-ready data is:
Unified
Real-time
Human-verified
Governed
Without it, AI can confidently fail. With it? Reliable, scalable results.
📖 Read More
💡 More insights → Data Engineering Weekly
#AI #AIReady #DataEngineering
AI-ready data is:
Unified
Real-time
Human-verified
Governed
Without it, AI can confidently fail. With it? Reliable, scalable results.
📖 Read More
💡 More insights → Data Engineering Weekly
#AI #AIReady #DataEngineering
September 23, 2025 at 8:56 AM
🚀 AI adoption is booming—but most data isn’t ready!
AI-ready data is:
Unified
Real-time
Human-verified
Governed
Without it, AI can confidently fail. With it? Reliable, scalable results.
📖 Read More
💡 More insights → Data Engineering Weekly
#AI #AIReady #DataEngineering
AI-ready data is:
Unified
Real-time
Human-verified
Governed
Without it, AI can confidently fail. With it? Reliable, scalable results.
📖 Read More
💡 More insights → Data Engineering Weekly
#AI #AIReady #DataEngineering
Stripe’s Real-Time Billing Analytics ⚡
Content:
Stripe wanted real-time visibility into subscriptions.
Traditional batch systems weren’t fast enough. ⏱️
They built a pipeline using Flink, Spark, and Pinot v2.
Now, analytics arrive in minutes, not hours. Queries return in <300ms. 🚀
Content:
Stripe wanted real-time visibility into subscriptions.
Traditional batch systems weren’t fast enough. ⏱️
They built a pipeline using Flink, Spark, and Pinot v2.
Now, analytics arrive in minutes, not hours. Queries return in <300ms. 🚀
September 22, 2025 at 1:00 PM
Stripe’s Real-Time Billing Analytics ⚡
Content:
Stripe wanted real-time visibility into subscriptions.
Traditional batch systems weren’t fast enough. ⏱️
They built a pipeline using Flink, Spark, and Pinot v2.
Now, analytics arrive in minutes, not hours. Queries return in <300ms. 🚀
Content:
Stripe wanted real-time visibility into subscriptions.
Traditional batch systems weren’t fast enough. ⏱️
They built a pipeline using Flink, Spark, and Pinot v2.
Now, analytics arrive in minutes, not hours. Queries return in <300ms. 🚀
The 238th edition of Data Engineering Weekly is available, featuring exciting Data & AI articles.
Read more:
www.dataengineeringw...
Read more:
www.dataengineeringw...
September 22, 2025 at 2:17 AM
The 238th edition of Data Engineering Weekly is available, featuring exciting Data & AI articles.
Read more:
www.dataengineeringw...
Read more:
www.dataengineeringw...
Apache Iceberg is now entering the classic paradox.
Reference:
www.dataengineeringw...
www.warpstream.com/b...
Reference:
www.dataengineeringw...
www.warpstream.com/b...
September 18, 2025 at 3:28 AM
Apache Iceberg is now entering the classic paradox.
Reference:
www.dataengineeringw...
www.warpstream.com/b...
Reference:
www.dataengineeringw...
www.warpstream.com/b...
From Firefighting to Proactive DB Reliability 🚀
Databricks engineers used Databricks to revolutionize DB reliability:
Query/Schema Scorer in CI pipelines
Delta Tables + DLT pipelines
Database Usage Scorecard across thousands of DBs
Efficiency ✅ Anti-patterns ❌
Databricks engineers used Databricks to revolutionize DB reliability:
Query/Schema Scorer in CI pipelines
Delta Tables + DLT pipelines
Database Usage Scorecard across thousands of DBs
Efficiency ✅ Anti-patterns ❌
September 16, 2025 at 1:01 PM
From Firefighting to Proactive DB Reliability 🚀
Databricks engineers used Databricks to revolutionize DB reliability:
Query/Schema Scorer in CI pipelines
Delta Tables + DLT pipelines
Database Usage Scorecard across thousands of DBs
Efficiency ✅ Anti-patterns ❌
Databricks engineers used Databricks to revolutionize DB reliability:
Query/Schema Scorer in CI pipelines
Delta Tables + DLT pipelines
Database Usage Scorecard across thousands of DBs
Efficiency ✅ Anti-patterns ❌
Parquet paradox: supports pluggable indexing and bloom filters, but you must rewrite entire files to use them. Meanwhile, LanceDB rebuilds indexes independently. Is "self-contained" showing off its age to "composable" data architectures? 🤔
September 9, 2025 at 7:58 PM
Parquet paradox: supports pluggable indexing and bloom filters, but you must rewrite entire files to use them. Meanwhile, LanceDB rebuilds indexes independently. Is "self-contained" showing off its age to "composable" data architectures? 🤔