Release DataFusion 45.0.0

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Last release was https://crates.io/crates/datafusion/440.0 December 31, 2024 so next major release would be around Feb 1, 2025

Steps:

• Update version and changelog: Prepare for 45.0.0 release: Version and Changelog #14397
• Voting Thread: https://lists.apache.org/thread/g20ywc9yto8xp07lcllmvgyn8g5z4420
• Release to crates.io
• Create ticket for next release: Release DataFusion 46.0.0 #14123
• Port changelog to main: [branch-45] Update changelog for backports to 45.0.0 #14460

Pre-relese testing

• Test with Comet: Test DataFusion 45.0.0 with Comet #14274
• Test with delta.rs: chore: update to DataFusion 45.0.0 / arrow/parquet 54.0.0 delta-io/delta-rs#3175
• Test with SailHQ: Test DataFusion 45.0.0 with Sail #14408
• Test with InfluxData
• Test with DataFusion Python: Test DataFusion 45 with datafusion-python #14410

Prior release tickets:

• 44.0.0 Release DataFusion 44.0.0 #13334

Please let me know if you would like to add any items on this list or move the categorization

Items to fix before release

• Upgrade to arrow-rs/parquet 54.0.0 #14114
• Regression: DataFrame::schema returns incorrect schema for NATURAL JOIN #14058
• Regression in CASE expression since DF 44 #14277
• string_numeric_coercion in binary.rs does not support Utf8View #13359
• DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (X --> String) for Specified UDFs #14230
• Regression: Invalid comparison operation: Utf8 == Utf8View error during LEFT ANTI JOIN #13510
• Type Coercion fails for List with inner type struct which has large/view types #14154
• [Epic] A Collection of Additional UTF8View support tickets #13504
• encode(..., "hex") errors on non-UTF-8 binaries since Datafusion v43 #14055
• Improve deprecation message for MemoryExec #14322
• LimitPushdown rule uncorrect remove some GlobalLimitExec #14204

Items maybe to complete (not sure if they are blockers)

• Can no longer easily join duplicate schemas as of version 43 #14112
• Query produces different results after physical plan round trip to bytes #14253
• Potential performance regression with comparisions to scalar values #14291
• Error with type coercion with CREATE TABLE AS SELECT ... inserting VALUES  #13124
• Bug: applying multiple times EnforceDistribution generates invalid plan #14150

Nice to Have (but non blockers -- e.g. bugs but not regressions)

• feat: metadata columns #14057
• Exponential planning time (100s of seconds) with UNION and ORDER BY queries #13748
• External Error prefix is repeated multiple times #14080
• FFI support for versions and alternate tokio runtimes #13937
• Improve speed of median by implementing special GroupsAccumulator  #13681
• FULL OUTER JOIN and LIMIT produces wrong results #14335

[alamb]

@andygrove would you like to coordinate this release or would you like me to? (or does anyone else want to do so?)

[alamb]

I also added some issues to the description above that I think would be worth fixing

[andygrove]

@andygrove would you like to coordinate this release or would you like me to? (or does anyone else want to do so?)

I don't have a preference. I will traveling around this time though, so perhaps it would make sense for someone else to be release manager for this one.

[alamb]

I don't have a preference. I will traveling around this time though, so perhaps it would make sense for someone else to be release manager for this one.

I am happy to do it again for 45 if no one else would like the opportunity (see what I did there 😆 )

[xudong963]

I don't have a preference. I will traveling around this time though, so perhaps it would make sense for someone else to be release manager for this one.

I am happy to do it again for 45 if no one else would like the opportunity (see what I did there 😆 )

Thanks, alamb, I booked 46 in advance!

[alamb]

I am happy to do it again for 45 if no one else would like the opportunity (see what I did there 😆 )

Thanks, alamb, I booked 46 in advance!

Awesome -- I filed #14123 to track 46

[alamb]

I plan to start assembing the release candidate and test on the week of Jan 27 (in about 2 weeks time()

[shehabgamin]

As promised, Sail is working on porting relevant tests into DataFusion.

A good starting point is a regression our tests caught in DataFusion 43, which still seems to persist in DataFusion 44. A regression was introduced in DataFusion 43.0.0 related to casting to UTF8 in various places. Upgrading to DataFusion 43.0.0 required adding explicit casting in several areas as a workaround. This PR (lakehq/sail#355) comments out those changes to expose the regression through the 12 additional failed tests compared to the main branch.

Once I’ve pinpointed the root cause(s) of the regression, I’ll create an issue in DataFusion to track the work. I want to ensure the issue accurately reflects the problem before filing it. I’m happy to address these regressions and port over the tests that cover them in the same PR. Hopefully, we can get this resolved in time for the DataFusion 45 release!

[alamb]

Once I’ve pinpointed the root cause(s) of the regression, I’ll create an issue in DataFusion to track the work. I want to ensure the issue accurately reflects the problem before filing it. I’m happy to address these regressions and port over the tests that cover them in the same PR. Hopefully, we can get this resolved in time for the DataFusion 45 release!

Thank you very much @shehabgamin 🙏

I strongly suspect this is related to switching to Utf8View by default in Parquet; You can validate this theory by disabling the following config setting:

https://datafusion.apache.org/user-guide/configs.html

datafusion.execution.parquet.schema_force_view_types

I think we are pretty close to closing out the Utf8View epic (now that we have upgraded to the latest arrow):

• [Epic] A Collection of Additional UTF8View support tickets #13504

I'll add that to the list for 45 too

[alamb]

I plan to start preparing / testing / pushing this release the week of Jan 27, aiming to get an release candidate early the next week

[shehabgamin]

I strongly suspect this is related to switching to Utf8View by default in Parquet; You can validate this theory by disabling the following config setting:

https://datafusion.apache.org/user-guide/configs.html

datafusion.execution.parquet.schema_force_view_types

I think we are pretty close to closing out the Utf8View epic (now that we have upgraded to the latest arrow):

* [[Epic] A Collection of Additional UTF8View support tickets #13504](https://github.com/apache/datafusion/issues/13504)

Thanks for the pointer @alamb!

I tried setting datafusion.execution.parquet.schema_force_view_types to false, but unfortunately, none of the 12 failed tests passed.

I'll take a deeper look into the issue after the weekend. Hope you have a great rest of your weekend!

[shehabgamin]

I'll take a deeper look into the issue after the weekend. Hope you have a great rest of your weekend!

Most of the regressions are related to this issue: #14230. I should be able to resolve them well before the 45 release.

While testing my local Sail code with the latest commit on DataFusion's main branch, I encountered several breaking changes that may make DataFusion 45 a jarring upgrade for some users. Given the previous discussion about wanting to make releases less jarring (#13334 (comment)), I wanted to bring this to your attention, @alamb.

Aside from that, there is one remaining regression I haven't investigated yet, which seems to be related to Parquet.

[alamb]

While testing my local Sail code with the latest commit on DataFusion's main branch, I encountered several breaking changes that may make DataFusion 45 a jarring upgrade for some users

Thanks @shehabgamin -- Can you enumerate these changes (or point me at a PR) so we can see if there is some way to make jarring

[shehabgamin]

Thanks @shehabgamin -- Can you enumerate these changes (or point me at a PR) so we can see if there is some way to make jarring

Yeah I'll work on that right now!

[shehabgamin]

My apologies @alamb, the DataFusion upgrade from the latest main branch commit is smoother than I initially thought. After investigating the flood of errors, I discovered that many were resolved by simply updating Sail's serde-arrow dependency to Arrow 54. Projects without PyO3 or the pyarrow feature in DataFusion should experience a seamless upgrade (as of writing). Projects using PyO3 with the pyarrow feature enabled will have varying experiences based on their usage of PyO3.

PyO3 0.23.3
DataFusion 45 upgrades from PyO3 0.22 to 0.23.3. This is an exciting change, but may introduce significant breaking changes for PyO3 users. Since these changes vary based on PyO3 usage, I'm not listing Sail's specific changes here. Users can refer to the PyO3 migration guide: https://pyo3.rs/v0.23.0/migration

DataFusion
ValuesExec is now deprecated. The deprecation message is a bit confusing though. It currently states: "Use MemoryExec::try_new_as_values instead", but I think should say: "Use MemoryExec::try_new_as_values or MemoryExec::try_new_from_batches instead". Or, just simply: "Use MemoryExec instead".

If you'd like to see these changes, they're in my PR that's testing the regression fixes: lakehq/sail#355

[jayzhan211]

ValuesExec is now deprecated. The deprecation message is a bit confusing though. It currently states: "Use MemoryExec::try_new_as_values instead", but I think should say: "Use MemoryExec::try_new_as_values or MemoryExec::try_new_from_batches instead". Or, just simply: "Use MemoryExec instead".

To replace ValuesExec, try_new_as_values is the right one to use not try_new_from_batches

[shehabgamin]

To replace ValuesExec, try_new_as_values is the right one to use not try_new_from_batches

Some people currently use ValuesExec::try_new_from_batches, so MemoryExec::try_new_as_values wouldn't necessarily be a suitable substitute.

[jayzhan211]

I see.

[Omega359]

I just upgraded my project to latest main from DF 42. The primary compilation and test suite issues I encountered after setting datafusion.execution.parquet.schema_force_view_types -> false (I'm going to fix all my .as_string_array(..) and utf8 datatype assumptions later):

utf8view, i32 comparison no longer worked

.with_column(
            STRING_FIELD,
            when(col(STRING_FIELD).eq(lit(83)), lit(82)).otherwise(col(STRING_FIELD))?,
        )?

switching to the obvious change below worked

.with_column(
            STRING_FIELD,
            when(col(STRING_FIELD).eq(lit("83")), lit("82")).otherwise(col(STRING_FIELD))?,
        )?

i32 -> utf8view coercion in regexp_like udf stopped working. For example: regexp_like(field_a, '[^0-9]') where field_a is an int field. changed to regexp_like(field_a::varchar, '[^0-9]') .. note, yes, I'm aware this is a really dumb test. I didn't write it though I will likely fix it.

Auto coercion of date/timestamp to utf8 no longer worked. I had to update my code to use to_char function to properly write out the date/timestamp. Not a bad thing really.

I'll likely do a full data run with this early next week.

[alamb]

Thanks @Omega359

The delta-rs upgrade seems to have gone pretty smootly: delta-io/delta-rs#3175

In terms of releasing DataFusion 45 I think it is better than 44 so I was planning to make the RC tomorrow, but I can wait a day or two if we want to make a push to finish up a few of those issues

utf8view, i32 comparison no longer worked

I believe this is fixed in the following PR (I am waiting on another committer to approve and if they do I can backport it)

• Support Utf8View to numeric coercion #14377

i32 -> utf8view coercion in regexp_like udf stopped working.

I believe this is tracked by

• Update REGEXP_MATCH scalar function to support Utf8View #11911

But is waiting on the next arrow release (which I just need to scrounge up one more PMC vote to approve and release). It might "just work" after that

• Release arrow-rs / parquet minor version 54.1.0 (Jan 2025) arrow-rs#6929

[kevinjqliu]

hey @alamb quick question about the release process, are the release for datafusion and datafusion-python in locksteps? I see the next release for datafusion is 45.0.0 meanwhile datafusion-python's main branch is currently on 43.0.0

is the plan to release datafusion 45.0.0 and then upgrade datafusion-python to 45.0.0 too and take on the new datafusion library version?

[shehabgamin]

Derived TPC-DS Query 66 (https://github.com/lakehq/sail/blob/main/python/pysail/data/tpcds/queries/q66.sql) fails on Sail, when it previously did not.

I'm out for the day, but will look more into this tonight.

[shehabgamin]

Derived TPC-DS Query 66 (https://github.com/lakehq/sail/blob/main/python/pysail/data/tpcds/queries/q66.sql) fails on Sail, when it previously did not.

Error is:

Exception: Error executing query q66 with error: Internal error:
Physical input schema should be the same as the one converted from logical input schema. 
Differences:
 - field nullability at index 7 [#98]: (physical) false vs (logical) true.

@alamb I created another PR that's dedicated only to the DF 45 upgrade. Could you please update the issue with this PR: lakehq/sail#365

Additionally, I created a tracking issue: #14408

[shehabgamin]

Another regression is that implementing the is_nullable function in the ScalarUDFImpl trait no longer works. For example:

impl ScalarUDFImpl for SparkArray {
    ...
    fn is_nullable(&self, _args: &[Expr], _schema: &dyn ExprSchema) -> bool {
        false
    }
    ...
}

Digging into the code, I see that it's deprecated (it should still work even though it's deprecated). What's strange, however, is that the deprecation warning is not propagating to me as a downstream user. I only found this out due to a failed test.

[alamb]

hey @alamb quick question about the release process, are the release for datafusion and datafusion-python in locksteps? I see the next release for datafusion is 45.0.0 meanwhile datafusion-python's main branch is currently on 43.0.0

Hi @kevinjqliu -- I am not sure what the plan for datafusion-python is

The main branch seems to be on 44 to me 🤔
https://github.com/apache/datafusion-python/blob/8b513906315a0749b9f5cd6f34bf259ab4dd1add/Cargo.toml#L41-L44

@timsaucer updated it a few weeks ago

• Chore/upgrade datafusion 44 datafusion-python#973

Any chance you can make a PR to test upgrading datafusion-python to 45?

Update: filed ticket to track:

• Test DataFusion 45 with datafusion-python #14410

[alamb]

Ok, given the feedback what I think we should do is:

1. Complete the testing on our release branch (branch-45) and backport any fixes needed
2. Continue development on main as previously planned.

Thanks @shehabgamin for the testing and reports. I'll take a more careful look tomorrow

[jayzhan211]

Digging into the code, I see that it's deprecated (it should still work even though it's deprecated). What's strange, however, is that the deprecation warning is not propagating to me as a downstream user. I only found this out due to a failed test.

I found that we can't make this deprecated but a breaking change if the user override the is_nullable function because it takes schema: &dyn ExprSchema which is not the case for return_type_from_exprs

[timsaucer]

Hi @kevinjqliu -- I am not sure what the plan for datafusion-python is

I am currently working on building the 44 release. I hope to get that done this morning or tomorrow. It's generally run about a month behind this upstream repo

[alamb]

Hi @kevinjqliu -- I am not sure what the plan for datafusion-python is

I am currently working on building the 44 release. I hope to get that done this morning or tomorrow. It's generally run about a month behind this upstream repo

In case anyone is interested, here is the release thread:

• https://lists.apache.org/thread/gkk4lq6k19gcq9xw64mcmbxrnf68o95s

[alamb]

Update on the release:

main is open for new code (will be 46.0.0)

I have created a 45.0.0 branch from which to make the 45.0.0 release

• https://github.com/apache/datafusion/tree/branch-45

I would very much like to get fixes for the following issues into the 45 branch as I think they are regressions that prevented some people from upgrading from 43 --> 44 and the fixes are fairly small:

• Do not rename struct fields when coercing types in CASE #14384
• Fix join type coercion when joining 2 relations with the same name via DataFrame API #14387 (regression from 43)
• Support Utf8View to numeric coercion #14377 (Utf8View regression in 43)
• Fix regression list Type Coercion List with inner type struct which has large/view types #14385 (regression from 43)
• Update REGEXP_MATCH scalar function to support Utf8View #11911 (Utf8View regression in 43) (mentioned by @Omega359 above)

It would also be really nice to sort out this issue from @shehabgamin but it isn't clear to me we can fix that with a small non-risky patch

• DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (X --> String) for Specified UDFs #14230

[alamb]

@Omega359 has a fix for #11911

• Update REGEXP_MATCH scalar function to support Utf8View #14449

This afternoon I will make some backport PRs and hopefully get a release candidate created

[alamb]

Backport PRs (for my own OCD sanity)

• [branch-45]: Backport chore: Upgrade to arrow/parquet 54.1.0 and fix clippy/ci (#14415) #14453
• [release-45] Fix join type coercion (#14387) #14454
• [branch-45] Support Utf8View to numeric coercion (#14377) #14455
• [branch-45] Fix regression list Type Coercion List with inner type struct which has large/view types (#14385) #14456
• [branch-45] Update REGEXP_MATCH scalar function to support Utf8View (#14449) #14457

[alamb]

Ok, I think we are ready with a release candidate:

tag here: https://github.com/apache/datafusion/releases/tag/45.0.0-rc1

Release voting thread: https://lists.apache.org/thread/g20ywc9yto8xp07lcllmvgyn8g5z4420

Content

---

This release candidate is based on commit: 26058ac 1
The proposed release tarball and signatures are hosted at 2.
The changelog is located at 3.

The standard verification procedure is documented at https://github.com/apache/datafusion/blob/main/dev/release/README.md#verifying-release-candidates.

[shehabgamin]

@alamb I'll test on Sail again sometime today!

[shehabgamin]

I updated #14408 (comment), but same issues as before with:
Test 2: Commit 26058ac (DataFusion 45.0.0-rc1).

[shehabgamin]

It would also be really nice to sort out this issue from @shehabgamin but it isn't clear to me we can fix that with a small non-risky patch

* [DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (Int --> String) #14230](https://github.com/apache/datafusion/issues/14230)

I think we came to a temporary solution here #14268 (comment)
CC @alamb @Omega359 @jayzhan211

[Omega359]

I don't think I'll have the bandwidth to test a type coercion fix for UDF's myself this week to be honest. I'm about to fire off a full run of my application against the 45 branch but I likely can only do that once this week.

[Omega359]

Beyond the type coercion issues that I can work around my testing is working ... as long as I don't compile with 'release' target. That seems to segfault I think somewhere in DF code but I haven't yet been able to get a core dump from the crashing nodes to investigate further. I thought it was Rust 1.84.1 dependent but I retested with 1.83.0 and had the same issue.

I don't think it's a blocker but is something I'm going to continue to try and narrow down.

[wiedld]

@alamb -- release candidate re-tested on InfluxData, and is good.