Ananth Packkildurai
@ananthdurai.bsky.social
Editor Data Engineering Weekly; subscribe www.dataengineeringweekly.com. In Prgress, LakeByte
The open source companies built their success on top of open-source platforms, benefited from community contributions and adoption, but now must abandon open-source principles to survive commercially.
November 10, 2025 at 2:47 AM
The open source companies built their success on top of open-source platforms, benefited from community contributions and adoption, but now must abandon open-source principles to survive commercially.
Cricket has been India’s greatest force in overcoming centuries of colonial suppression. Today’s Women’s World Cup win echoes the spirit of 1983 — a triumph that will inspire generations to come. 🇮🇳🏆
November 3, 2025 at 12:40 AM
Cricket has been India’s greatest force in overcoming centuries of colonial suppression. Today’s Women’s World Cup win echoes the spirit of 1983 — a triumph that will inspire generations to come. 🇮🇳🏆
Airbnb: Real-Time Key-Value Store
Airbnb’s next-gen key-value store supports real-time ingestion and bulk uploads with sub-second latency, powering feature stores and fraud detection.
Read the full story here: www.dataengineeringw...
Airbnb’s next-gen key-value store supports real-time ingestion and bulk uploads with sub-second latency, powering feature stores and fraud detection.
Read the full story here: www.dataengineeringw...
October 2, 2025 at 1:00 PM
Airbnb: Real-Time Key-Value Store
Airbnb’s next-gen key-value store supports real-time ingestion and bulk uploads with sub-second latency, powering feature stores and fraud detection.
Read the full story here: www.dataengineeringw...
Airbnb’s next-gen key-value store supports real-time ingestion and bulk uploads with sub-second latency, powering feature stores and fraud detection.
Read the full story here: www.dataengineeringw...
Grab: Partner Gateway Metrics at Sub-Second Speed
Real-time partner analytics at scale is tough. Grab uses Apache Pinot, Kafka–Flink ingestion, partitioning, and Star-tree indexing to cut query latency to <300 ms, enabling efficient API monitoring and fast issue resolution.
Real-time partner analytics at scale is tough. Grab uses Apache Pinot, Kafka–Flink ingestion, partitioning, and Star-tree indexing to cut query latency to <300 ms, enabling efficient API monitoring and fast issue resolution.
October 1, 2025 at 1:00 PM
Grab: Partner Gateway Metrics at Sub-Second Speed
Real-time partner analytics at scale is tough. Grab uses Apache Pinot, Kafka–Flink ingestion, partitioning, and Star-tree indexing to cut query latency to <300 ms, enabling efficient API monitoring and fast issue resolution.
Real-time partner analytics at scale is tough. Grab uses Apache Pinot, Kafka–Flink ingestion, partitioning, and Star-tree indexing to cut query latency to <300 ms, enabling efficient API monitoring and fast issue resolution.
Netflix Muse: Scaling Analytics at Trillion-Row Scale
Netflix evolved its Muse architecture to handle huge datasets efficiently: HyperLogLog sketches, Hollow in-memory feeds, and Druid optimizations cut query latency by ~50% and reduced concurrency load.
Netflix evolved its Muse architecture to handle huge datasets efficiently: HyperLogLog sketches, Hollow in-memory feeds, and Druid optimizations cut query latency by ~50% and reduced concurrency load.
September 30, 2025 at 12:33 PM
Netflix Muse: Scaling Analytics at Trillion-Row Scale
Netflix evolved its Muse architecture to handle huge datasets efficiently: HyperLogLog sketches, Hollow in-memory feeds, and Druid optimizations cut query latency by ~50% and reduced concurrency load.
Netflix evolved its Muse architecture to handle huge datasets efficiently: HyperLogLog sketches, Hollow in-memory feeds, and Druid optimizations cut query latency by ~50% and reduced concurrency load.
⚡ Latency Every Data Streaming Engineer Should Know
“Real-time” has limits—disk, network, and replication delays add up. StreamNative explains latency tiers, common costs, and tuning levers like batching & async processing.
💡 Must-read for data streaming engineers!
“Real-time” has limits—disk, network, and replication delays add up. StreamNative explains latency tiers, common costs, and tuning levers like batching & async processing.
💡 Must-read for data streaming engineers!
September 29, 2025 at 12:33 PM
⚡ Latency Every Data Streaming Engineer Should Know
“Real-time” has limits—disk, network, and replication delays add up. StreamNative explains latency tiers, common costs, and tuning levers like batching & async processing.
💡 Must-read for data streaming engineers!
“Real-time” has limits—disk, network, and replication delays add up. StreamNative explains latency tiers, common costs, and tuning levers like batching & async processing.
💡 Must-read for data streaming engineers!
MCP (Model Context Protocol) promises a new way for LLMs to use tools.
Chris Riccomini argues it mostly reinvents OpenAPI, gRPC & CLIs.
Resources = docs
Tools = RPC
Prompts = configs
So… could MCP have just been a JSON file?
💡 More insights: www.dataengineeringw...
Chris Riccomini argues it mostly reinvents OpenAPI, gRPC & CLIs.
Resources = docs
Tools = RPC
Prompts = configs
So… could MCP have just been a JSON file?
💡 More insights: www.dataengineeringw...
September 27, 2025 at 1:00 PM
MCP (Model Context Protocol) promises a new way for LLMs to use tools.
Chris Riccomini argues it mostly reinvents OpenAPI, gRPC & CLIs.
Resources = docs
Tools = RPC
Prompts = configs
So… could MCP have just been a JSON file?
💡 More insights: www.dataengineeringw...
Chris Riccomini argues it mostly reinvents OpenAPI, gRPC & CLIs.
Resources = docs
Tools = RPC
Prompts = configs
So… could MCP have just been a JSON file?
💡 More insights: www.dataengineeringw...
How Tables Got Smarter: Iceberg → DuckLake. From static snapshots to stream-native updates and catalog-first metadata, tables are evolving fast. Choose by intent, not hype.
Subscribe → www.dataengineeringw...
Full story → medium.com/fresha-da...
Subscribe → www.dataengineeringw...
Full story → medium.com/fresha-da...
September 26, 2025 at 12:33 PM
How Tables Got Smarter: Iceberg → DuckLake. From static snapshots to stream-native updates and catalog-first metadata, tables are evolving fast. Choose by intent, not hype.
Subscribe → www.dataengineeringw...
Full story → medium.com/fresha-da...
Subscribe → www.dataengineeringw...
Full story → medium.com/fresha-da...
How Tables Grew a Brain: Iceberg → DuckLake
Snapshots → incremental → stream-native → catalog-first.
Metadata is the bottleneck.
More insights → www.dataengineeringw...
Full story → medium.com/fresha-da...
Snapshots → incremental → stream-native → catalog-first.
Metadata is the bottleneck.
More insights → www.dataengineeringw...
Full story → medium.com/fresha-da...
September 25, 2025 at 12:33 PM
How Tables Grew a Brain: Iceberg → DuckLake
Snapshots → incremental → stream-native → catalog-first.
Metadata is the bottleneck.
More insights → www.dataengineeringw...
Full story → medium.com/fresha-da...
Snapshots → incremental → stream-native → catalog-first.
Metadata is the bottleneck.
More insights → www.dataengineeringw...
Full story → medium.com/fresha-da...
BlaBlaCar scales like a pro!
dbt Core → Transform like a champ
Airflow → Orchestrate effortlessly
CI/CD → Deploy instantly
Dev Containers → Standardized dev
📖 Full story →medium.com/blablacar...
💡 More insights → Subscribe to DEW
#DataEngineering #dbt #Airflow #CICD #DevContainers
dbt Core → Transform like a champ
Airflow → Orchestrate effortlessly
CI/CD → Deploy instantly
Dev Containers → Standardized dev
📖 Full story →medium.com/blablacar...
💡 More insights → Subscribe to DEW
#DataEngineering #dbt #Airflow #CICD #DevContainers
September 24, 2025 at 12:33 PM
BlaBlaCar scales like a pro!
dbt Core → Transform like a champ
Airflow → Orchestrate effortlessly
CI/CD → Deploy instantly
Dev Containers → Standardized dev
📖 Full story →medium.com/blablacar...
💡 More insights → Subscribe to DEW
#DataEngineering #dbt #Airflow #CICD #DevContainers
dbt Core → Transform like a champ
Airflow → Orchestrate effortlessly
CI/CD → Deploy instantly
Dev Containers → Standardized dev
📖 Full story →medium.com/blablacar...
💡 More insights → Subscribe to DEW
#DataEngineering #dbt #Airflow #CICD #DevContainers
🚀 AI adoption is booming—but most data isn’t ready!
AI-ready data is:
Unified
Real-time
Human-verified
Governed
Without it, AI can confidently fail. With it? Reliable, scalable results.
📖 Read More
💡 More insights → Data Engineering Weekly
#AI #AIReady #DataEngineering
AI-ready data is:
Unified
Real-time
Human-verified
Governed
Without it, AI can confidently fail. With it? Reliable, scalable results.
📖 Read More
💡 More insights → Data Engineering Weekly
#AI #AIReady #DataEngineering
September 23, 2025 at 8:56 AM
🚀 AI adoption is booming—but most data isn’t ready!
AI-ready data is:
Unified
Real-time
Human-verified
Governed
Without it, AI can confidently fail. With it? Reliable, scalable results.
📖 Read More
💡 More insights → Data Engineering Weekly
#AI #AIReady #DataEngineering
AI-ready data is:
Unified
Real-time
Human-verified
Governed
Without it, AI can confidently fail. With it? Reliable, scalable results.
📖 Read More
💡 More insights → Data Engineering Weekly
#AI #AIReady #DataEngineering
The 238th edition of Data Engineering Weekly is available, featuring exciting Data & AI articles.
Read more:
www.dataengineeringw...
Read more:
www.dataengineeringw...
September 22, 2025 at 2:17 AM
The 238th edition of Data Engineering Weekly is available, featuring exciting Data & AI articles.
Read more:
www.dataengineeringw...
Read more:
www.dataengineeringw...
Apache Iceberg is now entering the classic paradox.
Reference:
www.dataengineeringw...
www.warpstream.com/b...
Reference:
www.dataengineeringw...
www.warpstream.com/b...
September 18, 2025 at 3:28 AM
Apache Iceberg is now entering the classic paradox.
Reference:
www.dataengineeringw...
www.warpstream.com/b...
Reference:
www.dataengineeringw...
www.warpstream.com/b...
Parquet paradox: supports pluggable indexing and bloom filters, but you must rewrite entire files to use them. Meanwhile, LanceDB rebuilds indexes independently. Is "self-contained" showing off its age to "composable" data architectures? 🤔
September 9, 2025 at 7:58 PM
Parquet paradox: supports pluggable indexing and bloom filters, but you must rewrite entire files to use them. Meanwhile, LanceDB rebuilds indexes independently. Is "self-contained" showing off its age to "composable" data architectures? 🤔
AI Hallucinations = confident answers that are flat-out wrong.
Why they happen 👇
🎯 Training rewards sounding right, not being right
🎲 Guessing > “I don’t know”
📉 Missing data → confident fiction
The fix? Retrieval grounding + truth-focused training.
Why they happen 👇
🎯 Training rewards sounding right, not being right
🎲 Guessing > “I don’t know”
📉 Missing data → confident fiction
The fix? Retrieval grounding + truth-focused training.
September 9, 2025 at 1:00 PM
AI Hallucinations = confident answers that are flat-out wrong.
Why they happen 👇
🎯 Training rewards sounding right, not being right
🎲 Guessing > “I don’t know”
📉 Missing data → confident fiction
The fix? Retrieval grounding + truth-focused training.
Why they happen 👇
🎯 Training rewards sounding right, not being right
🎲 Guessing > “I don’t know”
📉 Missing data → confident fiction
The fix? Retrieval grounding + truth-focused training.
Building a Search Engine at Scale
3 B embeddings. 2 months. From content parsing to vector indexing. Wilson Lin shares how—and why chunking is modeling.
📖 www.dataengineeringw...
💡 Subscribe → www.dataengineeringw...
3 B embeddings. 2 months. From content parsing to vector indexing. Wilson Lin shares how—and why chunking is modeling.
📖 www.dataengineeringw...
💡 Subscribe → www.dataengineeringw...
August 29, 2025 at 1:00 PM
Building a Search Engine at Scale
3 B embeddings. 2 months. From content parsing to vector indexing. Wilson Lin shares how—and why chunking is modeling.
📖 www.dataengineeringw...
💡 Subscribe → www.dataengineeringw...
3 B embeddings. 2 months. From content parsing to vector indexing. Wilson Lin shares how—and why chunking is modeling.
📖 www.dataengineeringw...
💡 Subscribe → www.dataengineeringw...
Netflix is redefining data engineering.
With LanceDB + Media ML, the Lakehouse now powers media intelligence, not just metrics.
📖 netflixtechblog.com
💡 Subscribe: dataengineeringweekl...
With LanceDB + Media ML, the Lakehouse now powers media intelligence, not just metrics.
📖 netflixtechblog.com
💡 Subscribe: dataengineeringweekl...
August 28, 2025 at 1:00 PM
Netflix is redefining data engineering.
With LanceDB + Media ML, the Lakehouse now powers media intelligence, not just metrics.
📖 netflixtechblog.com
💡 Subscribe: dataengineeringweekl...
With LanceDB + Media ML, the Lakehouse now powers media intelligence, not just metrics.
📖 netflixtechblog.com
💡 Subscribe: dataengineeringweekl...
Reinforcement Learning Goes Enterprise
RLaaS is powering adaptive AI agents in the real world.
Dynamic AI that learns by doing = unstoppable. 🚀
📖 www.felicis.com/insi...
💡 More → www.dataengineeringw...
RLaaS is powering adaptive AI agents in the real world.
Dynamic AI that learns by doing = unstoppable. 🚀
📖 www.felicis.com/insi...
💡 More → www.dataengineeringw...
August 27, 2025 at 1:01 PM
Reinforcement Learning Goes Enterprise
RLaaS is powering adaptive AI agents in the real world.
Dynamic AI that learns by doing = unstoppable. 🚀
📖 www.felicis.com/insi...
💡 More → www.dataengineeringw...
RLaaS is powering adaptive AI agents in the real world.
Dynamic AI that learns by doing = unstoppable. 🚀
📖 www.felicis.com/insi...
💡 More → www.dataengineeringw...
⚡ Dagster: Streaming Workflows Done Right
From batch → event-first.
Dagster+ shows how to design reliable, observable real-time pipelines with Kafka & Flink.
Why wait for batch when your data can flow instantly?
📖 Full article → www.dataengineeringw...
From batch → event-first.
Dagster+ shows how to design reliable, observable real-time pipelines with Kafka & Flink.
Why wait for batch when your data can flow instantly?
📖 Full article → www.dataengineeringw...
August 26, 2025 at 1:00 PM
⚡ Dagster: Streaming Workflows Done Right
From batch → event-first.
Dagster+ shows how to design reliable, observable real-time pipelines with Kafka & Flink.
Why wait for batch when your data can flow instantly?
📖 Full article → www.dataengineeringw...
From batch → event-first.
Dagster+ shows how to design reliable, observable real-time pipelines with Kafka & Flink.
Why wait for batch when your data can flow instantly?
📖 Full article → www.dataengineeringw...
The 234th edition is out with the latest trends and thoughts in Data Engineering.
August 25, 2025 at 1:41 AM
The 234th edition is out with the latest trends and thoughts in Data Engineering.
🚀 LLMs: The Secret Superpower Behind Smarter Systems
LLMs aren’t just AI models—they’re system superheroes!
⚡ Execute code
🗄️ Pull context from databases
🌐 Surf live knowledge online
🧩 Solve complex problems like pros
LLMs aren’t just AI models—they’re system superheroes!
⚡ Execute code
🗄️ Pull context from databases
🌐 Surf live knowledge online
🧩 Solve complex problems like pros
August 23, 2025 at 1:00 PM
🚀 LLMs: The Secret Superpower Behind Smarter Systems
LLMs aren’t just AI models—they’re system superheroes!
⚡ Execute code
🗄️ Pull context from databases
🌐 Surf live knowledge online
🧩 Solve complex problems like pros
LLMs aren’t just AI models—they’re system superheroes!
⚡ Execute code
🗄️ Pull context from databases
🌐 Surf live knowledge online
🧩 Solve complex problems like pros
AI agents don’t just need training — they need the right context. ✨
From KV-cache optimization to external memory & error preservation, Manus shows how context engineering drives speed, recovery & scale.
📖 manus.im/blog/Contex...
💡 More deep dives → www.dataengineeringw...
From KV-cache optimization to external memory & error preservation, Manus shows how context engineering drives speed, recovery & scale.
📖 manus.im/blog/Contex...
💡 More deep dives → www.dataengineeringw...
August 22, 2025 at 1:00 PM
AI agents don’t just need training — they need the right context. ✨
From KV-cache optimization to external memory & error preservation, Manus shows how context engineering drives speed, recovery & scale.
📖 manus.im/blog/Contex...
💡 More deep dives → www.dataengineeringw...
From KV-cache optimization to external memory & error preservation, Manus shows how context engineering drives speed, recovery & scale.
📖 manus.im/blog/Contex...
💡 More deep dives → www.dataengineeringw...
Salesforce’s AIMS team shows how caching makes AI inference faster + more resilient 🚀
🔹 400ms → <1ms latency
🔹 27% faster requests
🔹 Survives DB outages
Scaling AI = speed + reliability.
📖 engineering.salesfor...
💡 www.dataengineeringw...
🔹 400ms → <1ms latency
🔹 27% faster requests
🔹 Survives DB outages
Scaling AI = speed + reliability.
📖 engineering.salesfor...
💡 www.dataengineeringw...
August 21, 2025 at 1:00 PM
Salesforce’s AIMS team shows how caching makes AI inference faster + more resilient 🚀
🔹 400ms → <1ms latency
🔹 27% faster requests
🔹 Survives DB outages
Scaling AI = speed + reliability.
📖 engineering.salesfor...
💡 www.dataengineeringw...
🔹 400ms → <1ms latency
🔹 27% faster requests
🔹 Survives DB outages
Scaling AI = speed + reliability.
📖 engineering.salesfor...
💡 www.dataengineeringw...
Meta Reinvents Compliance Management
Federation Platform → breaks down obligations into workstreams.
Privacy Waves → batches tasks monthly for predictability + accountability.
The goal: compliance that’s scalable and transparent.
📖 Full read: engineering.fb.com/2...
Federation Platform → breaks down obligations into workstreams.
Privacy Waves → batches tasks monthly for predictability + accountability.
The goal: compliance that’s scalable and transparent.
📖 Full read: engineering.fb.com/2...
August 20, 2025 at 1:00 PM
Meta Reinvents Compliance Management
Federation Platform → breaks down obligations into workstreams.
Privacy Waves → batches tasks monthly for predictability + accountability.
The goal: compliance that’s scalable and transparent.
📖 Full read: engineering.fb.com/2...
Federation Platform → breaks down obligations into workstreams.
Privacy Waves → batches tasks monthly for predictability + accountability.
The goal: compliance that’s scalable and transparent.
📖 Full read: engineering.fb.com/2...
Shopify MCP UI → Breaking the Text Wall
AI agents can now embed interactive UIs — product selectors, carts, image galleries — via MCP UI.
📌 Deep dive: shopify.engineering/...
💡 Also on DEW: www.dataengineeringw...
#AI #ShopifyEngineering #TechInnovation
AI agents can now embed interactive UIs — product selectors, carts, image galleries — via MCP UI.
📌 Deep dive: shopify.engineering/...
💡 Also on DEW: www.dataengineeringw...
#AI #ShopifyEngineering #TechInnovation
August 18, 2025 at 1:00 PM
Shopify MCP UI → Breaking the Text Wall
AI agents can now embed interactive UIs — product selectors, carts, image galleries — via MCP UI.
📌 Deep dive: shopify.engineering/...
💡 Also on DEW: www.dataengineeringw...
#AI #ShopifyEngineering #TechInnovation
AI agents can now embed interactive UIs — product selectors, carts, image galleries — via MCP UI.
📌 Deep dive: shopify.engineering/...
💡 Also on DEW: www.dataengineeringw...
#AI #ShopifyEngineering #TechInnovation