Summary
ObjectStoreRegistry.resolve() derives the object key from urlparse(url).path, which silently discards anything urlparse treats as URL params (;), query (?), or fragment (#). As a result the resolved key is truncated whenever a URL contains one of those characters.
This bites in two distinct, real situations:
Motivation 1 — object keys that contain # / ? / ;
These characters are valid in S3/GCS/Azure object keys. Concrete example: the Cell Painting Gallery (AWS Open Data) has image files named like DC_DCAM#1_CAM3.tif.
from obspec_utils.registry import ObjectStoreRegistry
from obstore.store import MemoryStore
reg = ObjectStoreRegistry()
reg.register("s3://bucket", MemoryStore())
store, path = reg.resolve("s3://bucket/path/DC_DCAM#1_CAM3.tif")
print(path) # -> 'path/DC_DCAM' ❌ expected 'path/DC_DCAM#1_CAM3.tif'
Downstream this surfaces as a spurious FileNotFound for the truncated key. We hit it via virtual_tiff → async-tiff reading the gallery:
AsyncTiffException: FileNotFoundError: Object at location .../Dest210531-152149/DC_DCAM ...
Motivation 2 — presigned URLs / SAS tokens for private buckets
A natural way to give a reader access to a private bucket without distributing credentials is to hand VirtualiZarr presigned URLs (or Azure SAS URLs), where the signature lives in the query string:
https://bucket.s3.us-east-1.amazonaws.com/key?X-Amz-Credential=...&X-Amz-Signature=...
resolve() currently drops the entire ?... query, i.e. it strips the signature, so every read against a generic HTTP store fails auth (403). Preserving the query is required for this pattern to work at all.
store, path = reg.resolve("https://host/data/file.nc?token=abc123")
print(path) # -> 'data/file.nc' ❌ token lost (expected 'data/file.nc?token=abc123')
Cause
registry.py, in resolve():
parsed = urlparse(url)
path = parsed.path # drops fragment ('#...'), query ('?...'), params (';...')
Fix
Reattach params/query/fragment to reconstruct the full key. PR to follow.
Note: for a presigned URL that resolves to an already-credentialed object store (rather than a
generic HTTP store), the reattached query would become part of the key — but that is a
contradictory configuration (the store already authenticates), and is out of scope for both
motivations above. Happy to gate on store type if maintainers prefer.
Summary
ObjectStoreRegistry.resolve()derives the object key fromurlparse(url).path, which silently discards anythingurlparsetreats as URL params (;), query (?), or fragment (#). As a result the resolved key is truncated whenever a URL contains one of those characters.This bites in two distinct, real situations:
Motivation 1 — object keys that contain
#/?/;These characters are valid in S3/GCS/Azure object keys. Concrete example: the Cell Painting Gallery (AWS Open Data) has image files named like
DC_DCAM#1_CAM3.tif.Downstream this surfaces as a spurious
FileNotFoundfor the truncated key. We hit it viavirtual_tiff→async-tiffreading the gallery:Motivation 2 — presigned URLs / SAS tokens for private buckets
A natural way to give a reader access to a private bucket without distributing credentials is to hand VirtualiZarr presigned URLs (or Azure SAS URLs), where the signature lives in the query string:
resolve()currently drops the entire?...query, i.e. it strips the signature, so every read against a generic HTTP store fails auth (403). Preserving the query is required for this pattern to work at all.Cause
registry.py, inresolve():Fix
Reattach params/query/fragment to reconstruct the full key. PR to follow.
Note: for a presigned URL that resolves to an already-credentialed object store (rather than a
generic HTTP store), the reattached query would become part of the key — but that is a
contradictory configuration (the store already authenticates), and is out of scope for both
motivations above. Happy to gate on store type if maintainers prefer.