[HUDI-9594] Allow Hudi to delegate catalog operations to Apache Polaris #13558

rahil-c · 2025-07-15T17:11:36Z

Change Logs

JIRA: https://issues.apache.org/jira/browse/HUDI-9594

This PR allows Hudi to integrate with Apache Polaris catalog by delegating createTable to the Polaris spark client, allowing hudi tables to be registered in the Polaris Catalog.
https://polaris.apache.org/in-dev/unreleased/polaris-spark-client/.

The key changes include:

Added Polaris detection logic in HoodieSqlCommonUtils.isUsingPolarisCatalog() to identify when Polaris catalog is configured in spark sessison.
Enhanced HoodieCatalog.createTable() to delegate table registration to Polaris after creating the Hudi table
Modified CreateHoodieTableCommand to skip Hive/SparkCatalog registration when Polaris is enabled
Introduced configurable Polaris class name via hoodie.datasource.polaris.catalog.class property
Add test with mock Polaris catalog to verify delegation behavior

Impact

Public API Changes:

New configuration property: hoodie.datasource.polaris.catalog.class (default: org.apache.polaris.spark.SparkCatalog)
Enhanced catalog behavior: When Polaris is detected, table creation delegates to external catalog

User-facing Changes:

Backward compatible: Existing deployments unaffected when Polaris not configured
polaris catalog experience: Hudi tables can be registered in Polaris catalog

Performance Impact:

minimal to no impact for create DDL

Risk level: Low

Verification done:

added unit tests covering both delegation and non-delegation scenarios
have tested with ITs in the Polaris repo
have tested integration end to end manually with local polaris catalog service

The risk is low because:

Delegation only occurs when explicitly configured with polaris spark catalog property in spark session

Documentation Update

Required updates:

Configuration reference: Document new hoodie.datasource.polaris.catalog.class property
Integration guide: Add Apache Polaris setup and configuration instructions

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

rahil-c · 2025-07-15T17:53:26Z

@yihua can you review this when you get a chance?

rahil-c · 2025-07-16T15:35:44Z

@Davis-Zhang-Onehouse if you can also take a look would be appreciated.

yihua

Production code changes LGTM.

...source/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlCommonUtils.scala

rahil-c · 2025-07-23T07:45:56Z

@yihua I have added a config for this polaris spark catalog class name in case the value changes in future. Please take a look when you get a chance

rahil-c · 2025-07-23T18:17:48Z

@yihua the ci is failing for the flaky test testRLIWithMDTCleaning. When i ran it locally it passes

wondering if we can just rerun this one failed job?

yihua · 2025-07-23T19:15:44Z

@yihua the ci is failing for the flaky test testRLIWithMDTCleaning. When i ran it locally it passes

wondering if we can just rerun this one failed job?

Yes. Only committers can retrigger a single failed job.

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala

yihua · 2025-07-24T13:17:55Z

hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala

+    .markAdvanced()
+    .withDocumentation("Fully qualified class name of the catalog that is used by the Polaris spark client.")


Suggested change

.markAdvanced()

.withDocumentation("Fully qualified class name of the catalog that is used by the Polaris spark client.")

.markAdvanced()

.sinceVersion("1.1.0")

.withDocumentation("Fully qualified class name of the catalog that is used by the Polaris spark client.")

Would it be better for me to specify this for the next point release, instead of major release. So it should be 1.0.3 instead of 1.1.0

Usually the new config is only added to major release. So it's preferred to use 1.1.0.

...source/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlCommonUtils.scala

yihua · 2025-07-24T13:26:32Z

...ce/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/catalog/MockPolarisSparkCatalog.scala

+ * Mock Polaris Spark Catalog for testing delegation behavior.
+ * Only implements essential methods: createTable and loadTable.
+ */
+class MockPolarisSparkCatalog extends TableCatalog {


Once the PR change to support Hudi in Polaris is merged, it would be good to add an integration tests using docker in GitHub action. Let's add a JIRA ticket to track that.

This is good idea, I am planning on adding integ tests in Polaris repo for hudi, but think we should make a followup to do same in hudi repo.

Filed the JIRA here: https://issues.apache.org/jira/browse/HUDI-9639

yihua · 2025-07-24T13:33:14Z

...rk/src/test/scala/org/apache/spark/sql/hudi/catalog/TestPolarisHoodieCatalogDelegation.scala

+    if (enablePolaris) {
+      hoodieCatalog.setDelegateCatalog(mockPolarisDelegate)
+    }


Does this mimic Polaris's Spark catalog behavior?

Yes, see this line here in the Polaris side changes https://github.com/apache/polaris/pull/1862/files#diff-4737e300174f8a433f92317317dc2cb6387148df6e8aee8e24742285e9feddc0R68

Sg, let's enhance the comment to mention this mimic Polaris's Spark catalog behavior

will do so thanks!

yihua

LGTM

hudi-bot · 2025-07-25T05:23:21Z

CI report:

b188880 UNKNOWN
462fd6f UNKNOWN
c41c691 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

rahil-c · 2025-07-25T05:26:27Z

@yihua was wondering if you can merge this when you get a chance?

github-actions bot added the size:L PR with lines of changes in (300, 1000] label Jul 15, 2025

rahil-c force-pushed the rahil/hoodie-catalog branch from 3786ba1 to 339e66c Compare July 16, 2025 01:09

yihua reviewed Jul 18, 2025

View reviewed changes

...source/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlCommonUtils.scala Outdated Show resolved Hide resolved

...source/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlCommonUtils.scala Show resolved Hide resolved

rahil-c force-pushed the rahil/hoodie-catalog branch from b188880 to 83a8ea2 Compare July 23, 2025 06:37

github-actions bot added size:M PR with lines of changes in (100, 300] and removed size:L PR with lines of changes in (300, 1000] labels Jul 23, 2025

rahil-c force-pushed the rahil/hoodie-catalog branch from 83a8ea2 to 462fd6f Compare July 23, 2025 07:44

rahil-c requested a review from yihua July 23, 2025 23:40

yihua reviewed Jul 24, 2025

View reviewed changes

...source/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlCommonUtils.scala Show resolved Hide resolved

yihua reviewed Jul 24, 2025

View reviewed changes

rahil-c requested a review from yihua July 24, 2025 18:41

yihua approved these changes Jul 24, 2025

View reviewed changes

rahil-c added 5 commits July 24, 2025 17:39

[HUDI-9594] Allow Hudi to delegate catalog operations to Apache Polaris

45992c7

make changes to have a config in case class name changes

ff8f078

scalastyle fix

4180d5d

address ethan comments

1003384

address ethan minor comments

c41c691

rahil-c force-pushed the rahil/hoodie-catalog branch from 40c455d to c41c691 Compare July 25, 2025 00:39

yihua merged commit 9f4ae4b into apache:master Jul 25, 2025
60 checks passed

rahil-c mentioned this pull request Jul 29, 2025

Initial integration for hudi tables within Polaris apache/polaris#1862

Open

		.markAdvanced()
		.withDocumentation("Fully qualified class name of the catalog that is used by the Polaris spark client.")

[HUDI-9594] Allow Hudi to delegate catalog operations to Apache Polaris #13558

[HUDI-9594] Allow Hudi to delegate catalog operations to Apache Polaris #13558

Conversation

rahil-c commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Risk level: Low

Documentation Update

Contributor's checklist

Uh oh!

rahil-c commented Jul 15, 2025

Uh oh!

rahil-c commented Jul 16, 2025

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rahil-c commented Jul 23, 2025

Uh oh!

rahil-c commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yihua commented Jul 23, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Jul 25, 2025

CI report:

Uh oh!

rahil-c commented Jul 25, 2025

Uh oh!

Uh oh!

Uh oh!

rahil-c commented Jul 15, 2025 •

edited

Loading

rahil-c commented Jul 23, 2025 •

edited

Loading