cc @julien.ledem.net the blog post is quite cool imo :)

It also speeds up files downloads and uploads, since now you only need to move the differentiating data around :)

find more about Xet here: huggingface.co/blog/xet-on-...

This writer outputs Parquet files that are robust to insertions/deletions/edits

Which means versioned datasets cost only a fraction of their original storage ! 🔥🤯

e.g. if you store with Xet, which deduplicates files by chunk

cc @julien.ledem.net FYI

you can define it this way:

>>> import pyarrow.parquet as pq
>>> writer = pq.ParquetWriter(
... out, schema,
... use_content_defined_chunking=True,
... )

Spreadsheet folk are welcome on the @hf.co hub too!

@lhoestq.hf.co
https://buff.ly/3VAEYKW