Add better S3 test cases to hubverse-transform #30
Assignees
Milestone
Comments
bsweger
added a commit
that referenced
this issue
Nov 14, 2024
When reading parquet files from S3, hubverse-transform does an initial read to get the schema (so we can override it if necessary). However, the read fails because it's reading the wrong thing, and the transform process tries to open the model-output data on the local filesystem instead of on S3. I opened an issue to address the lack of S3 test cases, which resulting in this bug hitting production: #30
bsweger
added a commit
that referenced
this issue
Nov 14, 2024
When reading parquet files from S3, hubverse-transform does an initial read to get the schema (so we can override it if necessary). However, the read fails because it's reading the wrong thing, and the transform process tries to open the model-output data on the local filesystem instead of on S3. I opened an issue to address the lack of S3 test cases, which resulting in this bug hitting production: #30
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We've hit a few errors when syncing hub files to S3 because our test suite isn't robust enough for S3-style file syncing.
Moto, my usual go-to for AWS mocking doesn't help because pyarrow's S3FS object isn't based on boto (it's possible that moto server could help)
We should do one of the following:
**Definition of Done
The text was updated successfully, but these errors were encountered: