While they still follow the logical model of Zarr, their physical layout is quite different, making them not entirely compatible with Zarr V3. That means a ZarrV3 with sharding cannot be understood by GDAL, is that correct? [2/2]

Hi @mdsumner.bsky.social, I've been wondering how GDAL will handle Zarr V3 extensions, specifically ZEP‑02 (sharding) and ZEP‑03 (variable chunking). Some tools, such as Zarrs, now support these extensions. [1/2]

Unfortunately, yes, perhaps this way it can support offset and length access of COG overviews, which is currently not supported by ZOG.

Thanks for the link, I noticed the zarrs creator is working on zarrs_ffi. I have almost zero knowledge of Rust code, but it looks quite promising! Hope this helps reduce friction. github.com/zarrs/zarrs_...

I don't think ZOG is a bad idea, I will not be surprised to see implementations of these in the future. [3/3] 🧵

3. Although it’s a valid COG/GeoTIFF, any GDAL operation (e.g., gdal_translate, gdalwarp, etc.) will strip out the appended chunk offset/length info. The only safe place to store extra bytes in a COG is between the IFH and the first IFD (where the ghost COG header currently resides). [2/3] 🧵

There are some issues with ZOG. Just in case you are interested in a bit of feedback.

1. It stores chunk offset and length information twice, which is redundant.

2. It won’t work with GDAL compression methods unsupported by numcodecs.

[1/3] 🧵

3) The comparisons they made Zarr vs COG were unfair. gitlab.eopf.copernicus.eu/cpm/eopf-cpm.... You can also check summary slide #28 here: sentinels.copernicus.eu/documents/d/.... [8/8] 🧵

I have recently asked a few questions and haven't received a clear answer either (related to Zarr shard) at github.com/EOPF-Sample-.... This is not a research exercise where nothing happens if we make a mistake; this will affect how we access data in the next decade. [7/N] 🧵

2) People are complaining about EOPF Zarr because the transition plan is unclear. Several valid points have been raised by j.anger and c.defranchis on the official forum, but to this day, they remain, in my opinion, unanswered, discourse.eopf.copernicus.eu. [6/N] 🧵

Every Zarr port must reimplement the core logic. I'm not one to say what is best for Zarr, but wouldn't it make more sense to develop a shared C library (like libhdf5) with a stable FFI? That would open Zarr to all EO ecosystems. [5/N] 🧵

If Zarr is the "future", what's the plan for non-Python users? Most EO/GIS software relies on GDAL, and while GDAL supports Zarr v3, none of the ZEPs > 1 are implemented yet, so ZOG won't work outside Python. Zarr's support outside Python is weak. [4/N] 🧵

1) Where is the C dynamic-link library? ... Unlike other fields, EO is quite diverse: the community uses MATLAB, R, Julia, IDL, or QGIS/ArcGIS, not just Python. I'm from Peru, and most people here use R over Python. These are not niche user bases. [3/N] 🧵

Some resistance to the new is understandable, especially considering how critical Copernicus data is->it helps save lives and influences the decision-making of the planet. That said, there are very valid reasons behind the pushback about Zarr. Here are a few that make sense. [2/N] 🧵

Interesting reading! As someone who has expressed concerns about the EOPF's adoption of Zarr, I think it's also fair to explain why there's some "pushback" from parts of the community about Zarr. [1/N] 🧵