Add asynchronous load method #10327
Draft
Add asynchronous load method #10327
+462
−56
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
topic-backends
topic-indexing
topic-documentation
topic-zarr
for more information, see https://pre-commit.ci
dcherian
reviewed
May 29, 2025
| @@ -490,6 +490,23 @@ def test_sub_array(self) -> None: | |||
| assert isinstance(child.array, indexing.NumpyIndexingAdapter) | |||
| assert isinstance(wrapped.array, indexing.LazilyIndexedArray) | |||
|
|
|||
| async def test_async_wrapper(self) -> None: | |||
dcherian
reviewed
May 29, 2025
xarray/tests/test_async.py
Outdated
| [ | ||
| ("sel", {"x": 2}), | ||
| ("sel", {"x": [2, 3]}), | ||
| ("sel", {"x": slice(2, 4)}), |
dcherian
reviewed
May 29, 2025
| print("inside LazilyVectorizedIndexedArray.async_get_duck_array") | ||
| from xarray.backends.common import BackendArray | ||
|
|
||
| if isinstance(self.array, BackendArray): |
There was a problem hiding this comment.
I think this is a lot cleaner. In my previous refactor, I was trying hard to not depend on BackendArray but that's unavoidable now for async stuff AFAICT
dcherian
reviewed
May 29, 2025
xarray/core/indexing.py
Outdated
| def get_duck_array(): | ||
| raise NotImplementedError | ||
|
|
||
| async def async_get_duck_array(): |
There was a problem hiding this comment.
80/20 on this. Alternatively, we special case IndexingAdapter inside async_get_duck_array which seems worse to me.
dcherian
reviewed
May 29, 2025
Comment on lines
+160
to
+162
| if isinstance(data, IndexingAdapter): | ||
| # These wrap in-memory arrays, and async isn't needed | ||
| return data.get_duck_array() |
There was a problem hiding this comment.
could be removed now that I added async_get_duck_array to the base class
Suggested change
| if isinstance(data, IndexingAdapter): | |
| # These wrap in-memory arrays, and async isn't needed | |
| return data.get_duck_array() |
dcherian
reviewed
May 29, 2025
| chunks=(5, 5), | ||
| dtype="f4", | ||
| dimension_names=["x", "y"], | ||
| attributes={"add_offset": 1, "scale_factor": 2}, |
There was a problem hiding this comment.
very important: now we test the decoding infra
dcherian
reviewed
May 30, 2025
| duck_array = await self.array.async_get_duck_array() | ||
| # ensure the array object is cached in-memory | ||
| self.array = as_indexable(duck_array) | ||
| return duck_array |
There was a problem hiding this comment.
Might need a deep copy here to match previous behavior
|
Notes to self:
|
for more information, see https://pre-commit.ci
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds an
.async_load()method toVariable, which works by plumbing asyncget_duck_arrayall the way down until it finally gets to the async methods zarr v3 exposes.Needs a lot of refactoring before it could be merged, but it works.
whats-new.rstapi.rstAPI:
Variable.load_asyncDataArray.load_asyncDataset.load_asyncDataTree.load_asyncload_dataset?load_dataarray?