Skip to content

Delta Sharing 1.0.0
Compare
Choose a tag to compare
@linzhou-db linzhou-db released this 15 Sep 00:01
· 104 commits to main since this release
v1.0.0
9b7c7c4
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

We are super excited to announce the release of Delta Sharing 1.0.0, which includes several exciting new features and artifacts.

Delta Format Sharing

In order to support Delta advanced features such as DeletionVectors and ColumnMapping in the Delta Sharing OSS Spark connector, “Delta format sharing” is introduced. The core Delta Sharing functionalities are grouped into the delta-sharing-client library and will be released separately. The response of Delta format sharing will not be processed by delta-sharing-client; instead,advanced Delta Spark library will be leveraged to read the data. We also upgraded the Delta Sharing server to support this. (#334, #335, #337, #338, #370, #374, #375, #376, #377, #378, #388, #389, #392, #393, #395)

JsonPredicate Enhancements (JsonPredicatesV2)

We enhanced JsonPredicates to support predicate-based filtering on all columns and not just partition columns. We also added support for the following schema types: Float, Double, Timestamp. (#349, #352, #368, #394)
We added JsonPredicate support in the pandas connector (#396)

Pagination on Delta Sharing Queries

Pagination on Delta Sharing data access APIs enables control over the number of files returned per Query Table request. This improves the scalability of the Delta Sharing service when working with tables with large file volumes, preventing timeouts and resource limit exceedances. (#352, #353, #354, #356, #360, #362, #366)

Other improvements and fixes

Support the release of Maven artifacts for both Scala 2.12 and Scala 2.13. #333
Support refresh token to refresh the pre-signed urls correctly for queries on the latest snapshot. (#385, #386, #387)
Fix bug where to_pandas doesn't properly handle the case when column names between Delta and parquet metadata are different in casing (#384)
A couple build and test fixes. (#330, #364, #369)
A couple fixes on the PROTOCOL description. #340, #365
Fix an issue in DeltaSharingSource: to double check sortedFetchedFiles contain the correct files and set previousOffset correctly. #328, #344

Credits

Abhijit Chakankar, Charlene Lyu, Kr eis, Lin Zhou, William Chau, andyl-db

Assets 6
Loading