feat: Implement ZEP 8 URL syntax support for zarr-python #3369
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add comprehensive ZEP 8 URL parsing and resolution system - Implement StoreAdapter ABC for extensible storage adapters - Add built-in adapters for file, memory, S3, GCS, HTTPS schemes - Support pipe-chained URLs like s3://bucket/data.zip|zip:|zarr3: - Add URLSegment parsing with validation - Integrate with zarr.open_group and zarr.open_array APIs - Include demo script and comprehensive test suite - Pass all existing tests + 35 new ZEP 8-specific tests
github-actions
bot
added
the
needs release notes
2 tasks
github-actions
bot
removed
the
needs release notes
…into feature/zep8-url-support
martindurant
reviewed
Sep 3, 2025
|
|
||
| @classmethod | ||
| def get_supported_schemes(cls) -> list[str]: | ||
| return ["s3"] |
There was a problem hiding this comment.
Historically, "s3a" has also existed. I think it was a spark/hadoop implementation detail (i.e,. another codebase/driver for the same storage backend), so it might not be relevant any more, but I thought I would mention it.
jbms
reviewed
Sep 3, 2025
| adapter_name = "gcs" | ||
|
|
||
| @classmethod | ||
| def get_supported_schemes(cls) -> list[str]: |
There was a problem hiding this comment.
It would be better, I think, to avoid supporting "gcs" in addition to "gs" since that will create implementation divergence.
jbms
reviewed
Sep 3, 2025
| return ["http", "https"] | ||
|
|
||
|
|
||
| class S3Adapter(RemoteAdapter): |
There was a problem hiding this comment.
Neuroglancer also supports "s3+http://endpoint/path" and "s3+https://endpoint/path" syntax to allow custom s3 endpoints (rather than AWS) to be used. The plain "s3://" syntax always refers to the real AWS endpoints, not a custom s3-compatible server.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements support for the ZEP 8 URL syntax in Zarr Python.
Some examples of what now works:
TODO:
docs/user-guide/*.rstchanges/closes #2943
fixes #2831
xref: zarr-developers/zeps#48
cc @jbms