v0.3.15

Changes

✨ New Features

• [FEAT] run cluster on commit @raunakab (#3461)
• [FEAT]: Support .clip function @conradsoon (#3136)
• [FEAT] Add cluster profiles @raunakab (#3426)
• [FEAT] add pyiceberg 0.8.0 support @rongfengliang (#3448)
• [FEAT] migrate schema inference → async, block at py boundary @andrewgazelka (#3432)
• [CHORE] connect: df.schema @andrewgazelka (#3353)
• [CHORE] connect test: df.get_attr @andrewgazelka (#3349)
• [FEAT] Get native execution enablement from DAFT_RUNNER @desmondcheongzx (#3409)
• [FEAT] Add ability to download log files from ray-cluster @raunakab (#3406)
• [FEAT] Add ability to run arbitrary command on a set working directory @raunakab (#3404)
• [FEAT] Add steps to spin up, submit job, and spin down ray clusters @raunakab (#3403)
• [CHORE] connect: add tests for df.take() method @andrewgazelka (#3385)
• [FEAT] Create new run workflow @raunakab (#3402)
• [FEAT] Enable group by keys in aggregation expressions @kevinzwang (#3399)
• [FEAT] Build release python wheels and upload to AWS S3 @raunakab (#3398)
• [FEAT] connect: Add support for select @andrewgazelka (#3344)
• [FEAT] connect: add df.limit and df.first @andrewgazelka (#3309)
• [FEAT] connect: to_daft_* use ref instead of value @andrewgazelka (#3355)
• [FEAT] connect: add alias support @andrewgazelka (#3342)
• [FEAT] Filter predicates in SQL join @kevinzwang (#3371)
• [FEAT] connect: collect @andrewgazelka (#3326)

🚀 Performance Improvements

• [PERF] Improve hash table probe side decisions for Swordfish @desmondcheongzx (#3327)

👾 Bug Fixes

• [BUG] Fix extension type display @jaychia (#3456)
• [BUG] Remove enum imports from match statements @raunakab (#3436)
• [BUG] Explicitly set IO config in unity catalog load table @colin-ho (#3453)
• [BUG] Include storage options in lance write commit @colin-ho (#3451)
• [BUG] Replace semicolons in filenames with underscore @raunakab (#3430)
• [BUG] Terminate nodes instead of stopping them @raunakab (#3427)
• [BUG] Fix run-cluster passing in environment variables wrongly @jaychia (#3422)

📖 Documentation

• [FEAT]: Support .clip function @conradsoon (#3136)
• [DOCS] Shorten union of Literals @desmondcheongzx (#3449)
• [DOCS] Add missing list expression entries @desmondcheongzx (#3428)

🧰 Maintenance

• [CHORE] Add warning in PyRunner to switch to Native @colin-ho (#3472)
• [CHORE] Address comments on previous PR @raunakab (#3473)
• [CHORE] Write tpch parquet files one at a time @colin-ho (#3396)
• [CHORE] Remove CountMode and ResourceRequest from public API @desmondcheongzx (#3429)
• [CHORE] Add schemas for remaining local plan ops @colin-ho (#3446)
• [CHORE] Put empty table when building probe table @colin-ho (#3445)
• [CHORE] Explain block_on function in common-runtime @colin-ho (#3442)
• [CHORE] connect: df.schema @andrewgazelka (#3353)
• [CHORE] Update execution config to turn on Ray tracing @jaychia (#3431)
• [CHORE] connect test: df.get_attr @andrewgazelka (#3349)
• [CHORE] Cleanup ExprResolver @kevinzwang (#3401)
• [CHORE] connect: add tests for df.take() method @andrewgazelka (#3385)
• [CHORE] Change IOConfig to be serialized into binary instead of JSON @kevinzwang (#3400)
• [CHORE] Pin PyIceberg version to <0.8 @kevinzwang (#3391)
• [CHORE] Add TPC-H queries in SQL @kevinzwang (#3392)
• [CHORE] connect: Optimize plans in connect @colin-ho (#3378)
• [CHORE] delete empty file xyz @andrewgazelka (#3370)

⬆️ Dependencies

14 changes

• Bump orjson from 3.10.11 to 3.10.12 @dependabot (#3464)
• Bump grpcio from 1.67.0 to 1.68.1 @dependabot (#3465)
• Bump arrow-buffer from 51.0.0 to 53.3.0 @dependabot (#3467)
• Bump regex-syntax from 0.7.5 to 0.8.4 @dependabot (#3468)
• Bump memmap2 from 0.9.4 to 0.9.5 @dependabot (#3470)
• Bump image from 0.25.4 to 0.25.5 @dependabot (#3471)
• Bump bytes from 1.7.1 to 1.8.0 @dependabot (#3411)
• Bump astral-sh/setup-uv from 3 to 4 @dependabot (#3410)
• Bump serde_json from 1.0.124 to 1.0.133 @dependabot (#3413)
• Bump sample-arrow2 from 0.1.0 to 0.17.2 @dependabot (#3414)
• Bump chrono-tz from 0.8.6 to 0.10.0 @dependabot (#3415)
• Bump azure-storage-blob from 12.17.0 to 12.24.0 @dependabot (#3416)
• Bump opencv-python from 4.8.1.78 to 4.10.0.84 @dependabot (#3417)
• Bump sqlalchemy from 2.0.25 to 2.0.36 @dependabot (#3418)