Test Modules
test_parquettest_file_based_datasource
Issue Description
Tests that read Parquet files intermittently fail with Arrow nightly builds, reporting that the Parquet magic bytes are not found in the footer, suggesting file corruption or that the file is not a valid Parquet file.
Root Cause
The exact cause is unclear, but the error occurs during PyArrow's Parquet reading:
pyarrow.lib.ArrowInvalid: Could not open Parquet input source '<Buffer>': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
One possible cause is that recent changes in PyArrow's Parquet reader might've introduced a bug.
Example Failure
pyarrow.lib.ArrowInvalid: Could not open Parquet input source '<Buffer>': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
File "/opt/miniforge/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 1413, in __init__
[fragment], schema=schema or fragment.physical_schema,
Proposed Fix
Investigate whether anything related to Parquet reading has changed in PyArrow recently. If this is an issue with Arrow, we should open a ticket and ensure it's fixed upstream.
Additional Context
This is specifically related to Arrow nightly builds.
Test Modules
test_parquettest_file_based_datasourceIssue Description
Tests that read Parquet files intermittently fail with Arrow nightly builds, reporting that the Parquet magic bytes are not found in the footer, suggesting file corruption or that the file is not a valid Parquet file.
Root Cause
The exact cause is unclear, but the error occurs during PyArrow's Parquet reading:
One possible cause is that recent changes in PyArrow's Parquet reader might've introduced a bug.
Example Failure
Proposed Fix
Investigate whether anything related to Parquet reading has changed in PyArrow recently. If this is an issue with Arrow, we should open a ticket and ensure it's fixed upstream.
Additional Context
This is specifically related to Arrow nightly builds.