`open_virtual()` by betolink · Pull Request #1348 · earthaccess-dev/earthaccess

betolink · 2026-05-18T16:36:00Z

This PR adds open_virtual() a method that simplifies how we open virtual stores behind EDL authentication.

We can open virtual collections listed at the collection level metadata, for now following the convention adopted by JPL using the following entry in RelatedUrls

{
  "Description": "Virtual data set reference ",
  "URLContentType": "DistributionURL",
  "URL": "https://archive.podaac.earthdata.nasa.gov/../MUR-JPL-L4-GLOB-v4.1_combined-ref.json",
  "Type": "GET DATA",
  "Subtype": "VIRTUAL COLLECTION"
}

Note: We may have multiple virtual stores for any given collection so we anticipate this convention may change in the near future, for now the assumption is that each collection has at most one virtual representation.

open_virtual() has the following kwargs

uri: this can be a collection level result from collection = earthaccess.search_datasets()[0] a local path, or a publicly accessible reference URL. If we use a collection from earthaccess, we don't need to deal with the cumbersome auth scafolding for the underlaying libraries; in all cases we'll get an xarray dataset instance pointing to our virtual dataset.This can be kerchunk or Icechunk stores
access: this one informs earthaccess if the virtual store points to S3 buckets or HTTP, without inspecting the references we need to be explicit for now. values are direct or external (same as indirect)
storage_options: if we need custom authentication for the virtual store this works the same way xarray uses storage options, not needed for NASA data as it's automatic.
force_external: This is a handy flag to force reference translation, this is if the only available store points to S3 but we are accessing the data from outside AWS us-west-2 setting this to True will translate the URLs on the fly for out of region access.
load: if we should load the indexes in our dataset, if we do we can address the dataset as any other xarray dataset and perform operations like subsetting and aggregations, if not we get a virtualizarr dataset backed by ManifestArray instances, this is only useful if we plan to add more virtual references and update the cube.

The method signature is as follows:

ea.open_virtual(
    uri: 'str | Path | earthaccess.DataCollection',
    *,
    access: 'str' = 'indirect',
    storage_options: 'dict[str, Any] | None' = None,
    force_external: 'bool' = False,
    load: 'bool' = True,
    **kwargs: 'Any',
) -> 'xr.Dataset'

Another relevant aspect is that load=False is slower than load=True for now as an artifact of VirtualiZarr not parsing all the coordinates and variables with embedded values(inlined values). The last release of VirtualiZarr supports it but I ran into some issues. For now load=False requires a round trip to a local store for the references, this worked for all stores but adds latency for example with a JSON serialized store opening the references can take up to a couple minutes depending on internet speed. load=True is fast as we only load coords and lazy load references (chunk manifests). This will have to change for upcoming releases to improve speed.

Here is a notebook showing some the the capabilities for this new feature:
https://notebooksharing.space/view/d2eb24605b6fb6b011710d955052f67de77c38728afdbc48ccc9cc33db34ff57#displayOptions=

Another relevant topic is that Icechunk does not support out of region access for stores with authentication, if something like this PR gets mered we will have the whole set of data virtualization available and no further changes are required in earthaccess. earth-mover/icechunk#2143

Description

"Ready for review" checklist

Place this Pull Request (PR) in draft until it is ready for review (see below)
Please review our Pull Request Guide
Mark "ready for review" after following instructions in the guide

Merge checklist

PR title is descriptive
PR body contains links to related and resolved issues (e.g. closes #1)
If needed, CHANGELOG.md updated
If needed, docs and/or README.md updated
If needed, unit tests added (unsure how? see below!)
All checks passing (tip: comment pre-commit.ci autofix if pre-commit is failing)
At least one approval

Need help? We welcome contributions at every experience level. You don't have to
write tests alone — open your PR and ask for help. It's also fine to let GitHub run tests
for you, via Continuous Integration (CI),
instead of running them locally. If anything fails and you're not sure why, just
mention @earthaccess-dev/maintainers in a comment and we'll work with you!

📚 Documentation preview 📚: https://earthaccess--1348.org.readthedocs.build/en/1348/

… homogenization - open_virtual(): open kerchunk/parquet/json/icechunk stores from URI or DataCollection with load=True (kerchunk engine) or load=False (VirtualiZarr) - force_external: download refs, rewrite s3:// to https:// for external access - DataCollection.virtual_collection_url(), get_s3_credentials(), get_s3_filesystem() - homogenize_dataset_codec_level(): patch Zlib codec levels on ManifestArrays - virtualize() default access changed from direct to indirect - build_obstore_registry: remove bearer token from HTTPStore (proxies return 303 redirects) - Update deps: virtualizarr latest, icechunk, matplotlib, hvplot; remove pandas pin

github-actions · 2026-05-18T16:36:15Z

👈 Launch a binder notebook on this branch for commit 1ccd8bf

I will automatically update this comment whenever this PR is modified

👈 Launch a binder notebook on this branch for commit 56609bc

👈 Launch a binder notebook on this branch for commit 2a57863

👈 Launch a binder notebook on this branch for commit 2eeb773

👈 Launch a binder notebook on this branch for commit 338e434

👈 Launch a binder notebook on this branch for commit 1541b00

👈 Launch a binder notebook on this branch for commit 33a2d92

👈 Launch a binder notebook on this branch for commit 9873ada

👈 Launch a binder notebook on this branch for commit e616aba

👈 Launch a binder notebook on this branch for commit 3238c24

👈 Launch a binder notebook on this branch for commit de15c4b

👈 Launch a binder notebook on this branch for commit 8272649

👈 Launch a binder notebook on this branch for commit eb6421a

👈 Launch a binder notebook on this branch for commit 4a8e2dc

👈 Launch a binder notebook on this branch for commit 2603859

👈 Launch a binder notebook on this branch for commit 79e37db

👈 Launch a binder notebook on this branch for commit ca7a0ee

👈 Launch a binder notebook on this branch for commit a7927fd

👈 Launch a binder notebook on this branch for commit 91ee595

…n region access from gh actions

# Conflicts: # tests/unit/test_virtual.py

Mock get_s3_filesystem to avoid real network calls in CI. Suppress ruff SIM117 by combining nested with statements.

danielfromearth · 2026-05-26T18:46:52Z

+
+    Parameters:
+        uri: A ``DataCollection``, or a path/URI to the virtual store
+            (``.icechunk``, ``.parquet``, or ``.json``).


For the TEMPO month-long icechunk, it's a directory (see this example notebook for reading the current, experimental TEMPO icechunk), not a .icechunk file. Not sure if that's always the case or not for icechunks.

yeah good catch, this was just a convention but can be changed, I think another way of making sure it's an icechunk store is the protocol prefix, icechunk:// that the icechunk store seems to know how to handle. @maxrjones, do you know the canonical way for this?

I added a DistributionUrl to the metadata record for the TEMPO collection experimented with in that notebook (referenced in my last comment). The current code raises an error for that icechunk in S3 because it is a directory, without .icechunk suffix:

import earthaccess as ea ea.login() collection = ea.search_datasets(short_name="TEMPO_NO2_L3", version="V04") vds = ea.open_virtual(collection[0]) # raises the error

error:

--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[15], line 11 7 short_name="TEMPO_NO2_L3", 8 version="V04", 9 ) 10 ---> 11 vds = ea.open_virtual(collection[0]) 12 vds File [/srv/conda/envs/notebook/lib/python3.13/site-packages/earthaccess/virtual/core.py:757](https://openscapes.2i2c.cloud/srv/conda/envs/notebook/lib/python3.13/site-packages/earthaccess/virtual/core.py#line=756), in open_virtual(uri, access, storage_options, force_external, load, **kwargs) 752 if not _is_icechunk_uri(url) and not _is_kerchunk_uri(url): 753 msg = ( 754 f"Unrecognised virtual store URL in collection: {url}. " 755 "Expected a .icechunk, .parquet, or .json file." 756 ) --> 757 raise ValueError( 758 msg, 759 ) 760 else: 761 url = str(uri) ValueError: Unrecognised virtual store URL in collection: s3://asdc-prod-public/virtual-reference-docs/TEMPO_L3_V04_icechunk_for_202601. Expected a .icechunk, .parquet, or .json file.

Yeah we need to change how we validate the VDS URL. Should we come up with a convention? e.g. if any in the URL we match icechunk... and we are opening a VDS we assume icechunk. This is to avoid users having to guess the store.

This is the leading proposal for a canonical URL representation of Zarr/Icechunk: https://github.com/jbms/url-pipeline.

It's still a draft, but it's where the ecosystem is converging. It's implemented in TensorStore and Neuroglancer, with a zarr-python PR in progress (#3369). Earthmover's Zarrs.jl already uses it. Its icechunk: scheme directly covers our case. <url>|icechunk: addresses a repo root (no node), and the spec defines directory-based format auto-detection. I'll check with Ian about the status of the zarr-python PR. I'd avoid inventing yet another URL scheme (e.g. ic+http) when this exists.

danielfromearth · 2026-05-27T16:25:56Z

+
+    Parameters:
+        uri: A ``DataCollection``, or a path/URI to the virtual store
+            (``.icechunk``, ``.parquet``, or ``.json``).


I added a DistributionUrl to the metadata record for the TEMPO collection experimented with in that notebook (referenced in my last comment). The current code raises an error for that icechunk in S3 because it is a directory, without .icechunk suffix:

import earthaccess as ea ea.login() collection = ea.search_datasets(short_name="TEMPO_NO2_L3", version="V04") vds = ea.open_virtual(collection[0]) # raises the error

error:

--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[15], line 11 7 short_name="TEMPO_NO2_L3", 8 version="V04", 9 ) 10 ---> 11 vds = ea.open_virtual(collection[0]) 12 vds File [/srv/conda/envs/notebook/lib/python3.13/site-packages/earthaccess/virtual/core.py:757](https://openscapes.2i2c.cloud/srv/conda/envs/notebook/lib/python3.13/site-packages/earthaccess/virtual/core.py#line=756), in open_virtual(uri, access, storage_options, force_external, load, **kwargs) 752 if not _is_icechunk_uri(url) and not _is_kerchunk_uri(url): 753 msg = ( 754 f"Unrecognised virtual store URL in collection: {url}. " 755 "Expected a .icechunk, .parquet, or .json file." 756 ) --> 757 raise ValueError( 758 msg, 759 ) 760 else: 761 url = str(uri) ValueError: Unrecognised virtual store URL in collection: s3://asdc-prod-public/virtual-reference-docs/TEMPO_L3_V04_icechunk_for_202601. Expected a .icechunk, .parquet, or .json file.

betolink · 2026-06-16T15:35:33Z

+# ---------------------------------------------------------------------------
+
+
+def _is_icechunk_uri(uri: str) -> bool:


This needs to be refactored, should we just match icechunk in the URI? should we use ic+http protocol notation?

betolink · 2026-06-16T15:37:03Z

+    repo = icechunk.Repository.open(storage=storage)
+    session = repo.readonly_session("main")
+    store = session.store
+    return xr.open_zarr(store, **kwargs)


The reason to return xarray ds instead of the store is to have a unified API with kerchunk. Also what else can we do with a store if it's not opening it with xarray?

betolink · 2026-06-16T15:38:05Z

+    url: str,
+    access: str = "indirect",
+    **kwargs: Any,
+) -> xr.Dataset:


This superseeds the PR (draft) Julius started working #1135

betolink · 2026-06-24T16:27:40Z

After testing more, seems like we could go ahead with the current PR and update to more native use of virtualizarr once some of the following issues get addressed. Virtualizarr only supports fsspec for opening a kerchunk store, this is kerchunk to ManifestStore, (Kerchunk parsers) and we still have issues with decoding FillValue.

One thing we need to define is how CMR will encode the store type, Max suggested following the convention here: https://github.com/jbms/url-pipeline/blob/main/schemes/icechunk.md looks good, although my personal preference would be to use the protocol prefix like git e.g. {store_type}+{net_protocol}:// so one could have icechunk+https://some_url and even with branches git already handles it with @Branch. e.g. icechunk+s3://bucket/store@branch I'd say there is a parity between git and icechunk in terms of branches nodes etc. Unless I'm missing something. Regardless, we should (and will) follow what the zarr/icechunk community decides and not use a custom schema.

What do you guys think? @maxrjones @danielfromearth

betolink added 3 commits May 5, 2026 12:46

Add virtual API docs page, link virtual_data.ipynb tutorial in nav

cba74a3

update wording

1ccd8bf

betolink and others added 12 commits May 18, 2026 11:39

Merge branch 'main' into virtual-open

c780bab

fix lint

a9cf363

fix typing defs

019b1da

fix mypy for manifest patching

06f4fb7

override direct access for virtual stores, todo: find a way to test i…

bb16701

…n region access from gh actions

update warning rules

4d3b57d

Merge remote-tracking branch 'origin/main' into virtual-open

98b3183

# Conflicts: # tests/unit/test_virtual.py

fix ruff

da93349

pinning vz to > 2.x

4077ca2

Merge branch 'main' into virtual-open

823c144

parallelize virtualizarr tests

bfb5170

merging kerchunk into virtualizarr deps

56609bc

betolink marked this pull request as ready for review May 19, 2026 19:55

betolink marked this pull request as draft May 19, 2026 21:41

betolink added 2 commits May 19, 2026 20:45

update icechunk opener

2a57863

Fix test deprecation warning for store (earthaccess-dev#1351)

2eeb773

Mock get_s3_filesystem to avoid real network calls in CI. Suppress ruff SIM117 by combining nested with statements.

betolink marked this pull request as ready for review May 20, 2026 15:32

maxrjones mentioned this pull request May 21, 2026

Review earthaccess.open_virtual NASA-IMPACT/veda-odd#387

Open

danielfromearth reviewed May 26, 2026

View reviewed changes

Merge branch 'main' into virtual-open

338e434

betolink requested a review from chuckwondo May 27, 2026 15:03

danielfromearth requested changes May 27, 2026

View reviewed changes

betolink and others added 4 commits May 30, 2026 00:20

Merge branch 'main' into virtual-open

1541b00

Merge branch 'main' into virtual-open

33a2d92

Merge branch 'main' into virtual-open

9873ada

simplify uri resolution for virtual_open

e616aba

betolink added 4 commits June 10, 2026 09:40

upgrade VCRpy to v8.0.0

3238c24

fix linting

de15c4b

update min deps

8272649

update min deps again

5330b6f

betolink commented Jun 16, 2026

View reviewed changes

betolink added 2 commits June 23, 2026 12:40

update diagream

eb6421a

resolv merge conflict

4a8e2dc

betolink mentioned this pull request Jun 23, 2026

FillValue decoding unexpected fill_value: null (on version 2.6.1) zarr-developers/VirtualiZarr#989

Open

betolink and others added 5 commits July 1, 2026 12:12

Merge branch 'main' into virtual-open

2603859

fix lint

79e37db

return true if the url has icechunk on it

ca7a0ee

mock store for unit test

a7927fd

remove diagram from vds notebook

91ee595

betolink requested a review from danielfromearth July 1, 2026 21:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

`open_virtual()` #1348

`open_virtual()` #1348
betolink wants to merge 33 commits into
earthaccess-dev:mainfrom
betolink:virtual-open

betolink commented May 18, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 18, 2026 •

edited

Loading

Uh oh!

danielfromearth May 26, 2026

Uh oh!

betolink May 26, 2026

Uh oh!

danielfromearth May 27, 2026

Uh oh!

betolink May 27, 2026

Uh oh!

maxrjones Jun 16, 2026

Uh oh!

danielfromearth May 27, 2026

Uh oh!

betolink Jun 16, 2026

Uh oh!

betolink Jun 16, 2026

Uh oh!

betolink Jun 16, 2026

Uh oh!

betolink commented Jun 24, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		# ---------------------------------------------------------------------------


		def _is_icechunk_uri(uri: str) -> bool:

Uh oh!

Conversation

betolink commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

"Ready for review" checklist

Merge checklist

Uh oh!

github-actions Bot commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

betolink commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

betolink commented May 18, 2026 •

edited

Loading

github-actions Bot commented May 18, 2026 •

edited

Loading

betolink commented Jun 24, 2026 •

edited

Loading