It also speeds up files downloads and uploads, since now you only need to move the differentiating data around :)

find more about Xet here: huggingface.co/blog/xet-on-...

This writer outputs Parquet files that are robust to insertions/deletions/edits

Which means versioned datasets cost only a fraction of their original storage ! 🔥🤯

e.g. if you store with Xet, which deduplicates files by chunk

cc @julien.ledem.net FYI

CDC Parquet writer is out in PyArrow nightlies 🔥🔥

$ pip install \
-i pypi.anaconda.org/scientific-p... \
"pyarrow>=21.0.0.dev0"

it's changing the way I view data versioning👇