RADKLIM data reader implementation with vzarr and icechunk.

[wael-mika]

Is your feature request related to a problem? Please describe.

Implementing and customize a data reader for Radklim data set (stored in NetCDF format) based on virtualizarr and icechunk libraries.

The main issue is that current anemoi data sets are supported by zarr version =2.17 while icechunk requires zarr version = 3.xx

The data reader will be based on the base class with some customization to handle the parallaization issue related to reading from virtual icechunk store

Describe the solution you'd like

Data reader class based on the data reader class utilizing icechunk store as a reference to read the meta data and request the correct data from the correct netcdf file when called.

The data reader will adapt the base class and time window handler fuctionality and use lazy opening functionality to handle the multi worker setup of the project.

For the time being, this is an experimental feature to compare the perfomance with the kerchunk implementation and the subsequent errors when training.

A new environemnt must be included (does not include anemoi dependency) to avoide the clash with the anemoi data set,.

Describe alternatives you've considered

#386

Additional context

No response

Organisation

JSC

[wael-mika]

assign me

[github-actions]

Issue assigned to @wael-mika.

[wael-mika]

The dependency issue when adding icechunk library:

we can conclude that icechunk<0.2.3 depends on zarr>=3.
      And because icechunk>=0.2.3 depends on one of:
          zarr>=3,<3.0.3
          zarr>3.0.3
      and weathergen depends on icechunk, we can conclude that weathergen depends on zarr>=3.
      And because weathergen depends on zarr>=2.17,<3.dev0 and your workspace requires weathergen, we can conclude that your
      workspace's requirements are unsatisfiable.

The solution is to create a new and isolated environment to run and test the data reader with the required libraries until the testing of the Kerchunk implementation is done.

[wael-mika]

The data reader was tested using the sampler MultiStreamDataSampler to extract the data from the current implementation.
Date range: 2010-01-01T00:00:00 to 2010-01-15T00:00:00
Window: 12 hours, step 1 hour
Batch size: 2
Sampler instantiation:~1.3 seconds from start to sampler instantiation

2025-07-24 15:12:40,105 - INFO - Instantiating MultiStreamDataSampler with Radklim (icechunk) stream only...
2025-07-24 15:12:41,435 - INFO - Sampler instantiated successfully.

Lazy loading of Icechunk (Zarr) dataset
Batch fetch times (80–82 seconds per batch): Batch size: 2 samples.
No dataset or tokenization errors observed, only performance and chunking concerns.

[wael-mika]

Data reader impelmentation requires very specific settings and handling of the paths where the store, actual data and stats are stored:

 # ——— Paths & Stats ————————————————————————————————————————————————
        self.store_path = Path(stream_info["icechunk_store"])
        self.norm_path = Path(stream_info["stats_path"])
        self.netcdf_base_path = Path(stream_info["netcdf_base_path"])

        # ——— Programmatic Icechunk config + auth ————————————————————————
        self.url_prefix = f"file://{self.netcdf_base_path.resolve()}"
        self.repo_config = RepositoryConfig.default()
        vcont = VirtualChunkContainer(
            self.url_prefix,
            local_filesystem_store(str(self.netcdf_base_path.resolve()))
        )
        self.repo_config.set_virtual_chunk_container(vcont)
        self.auth = { self.url_prefix: None }

        # ——— Read metadata to get full time axis —————————————————————————
        storage = local_filesystem_storage(str(self.store_path))
        repo = Repository.open(
            storage,
            config=self.repo_config,
            authorize_virtual_chunk_access=self.auth
        )
        session = repo.readonly_session("main")
        with xr.open_zarr(session.store, consolidated=False) as ds_meta:
            self.times_full = ds_meta["time"].values.astype("datetime64[ns]")

[wael-mika]

No PR can be created here as the dependency problem is a main factor, and the performance of the reader is under study now to check for efficiency. However, scripts for creating the store reference, data reader implementation, and testing can be shared after discussion.