Skip to content

resolve() truncates object keys containing '#', '?', or ';' (breaks special-char keys and presigned URLs) #85

Description

@TomNicholas

Summary

ObjectStoreRegistry.resolve() derives the object key from urlparse(url).path, which silently discards anything urlparse treats as URL params (;), query (?), or fragment (#). As a result the resolved key is truncated whenever a URL contains one of those characters.

This bites in two distinct, real situations:

Motivation 1 — object keys that contain # / ? / ;

These characters are valid in S3/GCS/Azure object keys. Concrete example: the Cell Painting Gallery (AWS Open Data) has image files named like DC_DCAM#1_CAM3.tif.

from obspec_utils.registry import ObjectStoreRegistry
from obstore.store import MemoryStore

reg = ObjectStoreRegistry()
reg.register("s3://bucket", MemoryStore())

store, path = reg.resolve("s3://bucket/path/DC_DCAM#1_CAM3.tif")
print(path)   # -> 'path/DC_DCAM'   ❌  expected 'path/DC_DCAM#1_CAM3.tif'

Downstream this surfaces as a spurious FileNotFound for the truncated key. We hit it via virtual_tiffasync-tiff reading the gallery:

AsyncTiffException: FileNotFoundError: Object at location .../Dest210531-152149/DC_DCAM ...

Motivation 2 — presigned URLs / SAS tokens for private buckets

A natural way to give a reader access to a private bucket without distributing credentials is to hand VirtualiZarr presigned URLs (or Azure SAS URLs), where the signature lives in the query string:

https://bucket.s3.us-east-1.amazonaws.com/key?X-Amz-Credential=...&X-Amz-Signature=...

resolve() currently drops the entire ?... query, i.e. it strips the signature, so every read against a generic HTTP store fails auth (403). Preserving the query is required for this pattern to work at all.

store, path = reg.resolve("https://host/data/file.nc?token=abc123")
print(path)   # -> 'data/file.nc'   ❌  token lost  (expected 'data/file.nc?token=abc123')

Cause

registry.py, in resolve():

parsed = urlparse(url)
path = parsed.path          # drops fragment ('#...'), query ('?...'), params (';...')

Fix

Reattach params/query/fragment to reconstruct the full key. PR to follow.

Note: for a presigned URL that resolves to an already-credentialed object store (rather than a
generic HTTP store), the reattached query would become part of the key — but that is a
contradictory configuration (the store already authenticates), and is out of scope for both
motivations above. Happy to gate on store type if maintainers prefer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions