Update xarray version in Pangeo-EOSC environment to include datatree support

[tinaok]

Hi team 👋

I noticed that the current Pangeo-EOSC environment is using an older version of xarray which does not include support for datatree. This limits workflows that rely on hierarchical dataset structures.

Would it be possible to:
• bump to the latest released image:
👉 https://github.com/pangeo-data/pangeo-docker-images/releases/tag/2025.01.24
• or wait for this PR to be merged (which includes datatree):
👉 pangeo-data/pangeo-docker-images#603

??

[sebastian-luna-valero]

Hi Tina,

Currently, we are using 2024.08.18:

pangeo-eosc/providers/kubernetes/daskhub.yaml

Line 211 in 7a5c306

tag: 2024.08.18

Which points to xarray-datatree==0.0.14:
https://github.com/pangeo-data/pangeo-docker-images/blob/2024.08.18/pangeo-notebook/packages.txt#L705

Then the 2025.01.25 version of the pangeo-notebook image has xarray==2025.1.1 but not xarray-datatree:
https://github.com/pangeo-data/pangeo-docker-images/blob/2025.01.24/pangeo-notebook/packages.txt#L698

Is this because datatree has been added inside xarray?

On the other hand, it looks like pangeo-data/pangeo-docker-images#603 updates to xarray==2025.1.2
https://github.com/pangeo-data/pangeo-docker-images/blob/upgrade-pangeo-metapackage-2025.02.14/pangeo-notebook/packages.txt#L730

So could you please confirm whether you want the upgrade to version 2025.01.25 of the pangeo-notebook? or do you prefer to wait for the 2025.02.14 version to be release?

[scottyhq]

Is this because datatree has been added inside xarray?

Correct, datatree is part of xarray>=2024.10.0
https://github.com/pydata/xarray/blob/main/doc/whats-new.rst#v2024100-oct-24th-2024

[sebastian-luna-valero]

Thanks for confirming @scottyhq

@tinaok do you prefer that we update pangeo-notebook to 2025.01.25 asap? or should we wait for version 2025.02.14 via pangeo-data/pangeo-docker-images#603

[tinaok]

Thank you for pinging @sebastian-luna-valero, If it is a few days we wait, if not I would like that we have already the 2025.01.25

@scottyhq what's the time line for the version 2025.02.14?

[scottyhq]

No timeline - there is some discussion on the open PR on challenges with version incompatibilities so I'd suggest sticking with 2025.01.25 for a while, it as it has the datatree support you're after!

Also, for what it's worth I think given the fast development these days there are always issues that crop up so sitting on the same image for 3-6 months is probably a good strategy for a Hub (e.g. NCAR does every 6 months https://ncar-hpc-docs.readthedocs.io/en/latest/environment-and-software/user-environment/conda/#the-ncar-python-library)

Not sure how the Hub is configured, but 2i2c's 'bring-your-own image' selection is also really helpful for situations like these! https://2i2c.org/community-showcase/user/howto/specify-unlisted-image.html#specify-your-own-image-for-the-software-environment

[sebastian-luna-valero]

Thanks @scottyhq

I proposed #95 for @tinaok to review and approve.

The bring-your-own image selection looks very interesting! @tinaok is this of interest? (cc @annefou) if yes, I would like to see how to configure this. Do you know @scottyhq?

[scottyhq]

Do you know @scottyhq?

I do not! But 2i2c's infrastructure config is out in the open, so I'd try digging into there documentation, or maybe here https://github.com/2i2c-org/infrastructure

[tinaok]

The bring-your-own image selection looks very interesting! @tinaok is this of interest? (cc @annefou)

Yes it is very interesting! Would it work for dask worker's image?

[sebastian-luna-valero]

mmm, good point, I would say it won't, but if you are really interested, we could give it a go.

[sebastian-luna-valero]

Ok, I have created a follow up issue for the bring your own image option. I will test this at some point but I think there is no rush so scheduling it as low priority.

Since the container images have been updated, I will close this ticket.