Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pyspark DataFusion integration test #850

Merged
merged 3 commits into from
Jan 23, 2025

Conversation

gruuya
Copy link
Contributor

@gruuya gruuya commented Dec 26, 2024

This tests propagation of all supported primitive types in an e2e manner.

Alternative to #825.

@gruuya gruuya force-pushed the spark-datafusion-e2e branch from 6709685 to 318ff77 Compare December 27, 2024 08:26
kevinjqliu
kevinjqliu previously approved these changes Jan 4, 2025
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Great to see an example using datafusion's TableProvider

Copy link
Contributor

@sdd sdd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, once the Cargo.toml change is made. Thanks :-)

@gruuya gruuya force-pushed the spark-datafusion-e2e branch 2 times, most recently from abfd516 to b6e171e Compare January 17, 2025 09:35
Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kevinjqliu kevinjqliu requested a review from sdd January 17, 2025 17:05
Copy link
Contributor

@liurenjie1024 liurenjie1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gruuya for this pr, I prefer this one than #825.

This is so that both the datafusion integration and integration test crates can inherit the same version.
@gruuya gruuya force-pushed the spark-datafusion-e2e branch from 98cadea to be35fb1 Compare January 22, 2025 08:12
@gruuya
Copy link
Contributor Author

gruuya commented Jan 22, 2025

Seems like we're close to consensus that this is merge-worthy. I take it @sdd agrees, and I'm wondering are there any other objections? @Xuanwo

@sdd
Copy link
Contributor

sdd commented Jan 22, 2025

It's a yes from me @gruuya - approved 🙂

@Xuanwo
Copy link
Member

Xuanwo commented Jan 23, 2025

Thank you @gruuya for working on this, and thank you @Fokko, @kevinjqliu, @liurenjie1024 and @sdd for the review. Sorry for the late, let's move!

@Xuanwo Xuanwo merged commit ce068db into apache:main Jan 23, 2025
18 checks passed
@gruuya gruuya deleted the spark-datafusion-e2e branch January 23, 2025 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants