New blog post: Building a 0$ Data Distribution System.

juhache.substack.com/p/0-data-dis...

Some learnings after helping +50 companies in high performance data engineering projects

javisantana.com/2024/11/30/l...

Oh yeah

wiki.postgresql.org/wiki/Don%27t...

1 Docker container embedding app code + SQLite DB → live chat app with 10k simultaneous users.

youtu.be/0rlATWBNvMw

Yup, there are almost fifteen million SQLite databases on Bluesky’s PDS servers. It’s wildly efficient and simple but not without trade offs of course.

Makes sense for this use case in large part because each users atproto repository is self contained, with links to other repos, like a website.

Here's what I put in my ~/.zshrc file to make sure my virtualenv autoactivates when I move to a directory with a .venv file. Works well for me so far. Do the rest of you do something like this?

#Python #DataBS

Prediction: poeple will monetize custom feeds
github.com/bluesky-soci...

Something interesting is brewing in Iceberg-on-S3 land. 👀

lists.apache.org/thread/v7x65...

cc @eatonphil.bsky.social

Building a data pipeline =
50% bringing data from A to B at time t
50% making pipeline fixing easy at time t+1

New blog post: GCP & Iceberg

open.substack.com/pub/juhache/...

Now more than ever

Is there a starter pack for data engineering here on bluesky?

2010 — 2017:
ML = pip install scikit-learn

2017 — 2023:
ML = pip install torch

2023 — :
ML = pip install requests

FINALLY! 🎉
aws.amazon.com/blogs/comput...

No more endless searching for support with intrinsic
docs.aws.amazon.com/step-functio...

What about dumping Snowflake Marketplace free listings to R2 ?
@jakthom.bsky.social @ssp.sh @tobilg.com

The Snowflake unbundling continues:
• Snowflake Storage → Iceberg
• Snowflake Marketplace → Cloudflare R2 + DuckDB :)