feat(catalog): add catalog API #62

QuakeWang · 2024-08-22T06:44:58Z

fix: #52

Xuanwo · 2024-08-30T12:33:12Z

crates/paimon/src/catalog/mod.rs

+/// Impl References: <https://github.com/apache/paimon/blob/release-0.8.2/paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java#L42>
+#[async_trait]
+pub trait Catalog: Send + Sync {
+    const DEFAULT_DATABASE: &'static str = "default";


Using const will make this trait non-object safe, preventing users from invoking Catalog through Arc<dyn Catalog>. How about introducing a new API, such as info, for this purpose?

OK, I will do it later.

Xuanwo · 2024-08-30T12:34:04Z

crates/paimon/src/catalog/mod.rs

+    fn options(&self) -> &HashMap<String, String>;
+
+    /// Returns the FileIO instance.
+    fn file_io(&self) -> &FileIO;


The catalog is integrated with file I/O, which is somewhat surprising to me.

The catalog is integrated with file I/O, which is somewhat surprising to me.

The FileIO fileIO() function can be found at here. Do you have some good suggestions to deal this?

I need some input from @JingsongLi, @SteNicholas, and @Aitozi: Should a catalog be coupled with a specific file I/O?

Currently, the catalog is bind with a warehouse path which determines the fileIO. So these two things are coupled in current shape, and the table's schema are also retrieved from the filesystem. Also like to see some inputs from @JingsongLi @SteNicholas

@QuakeWang, @Xuanwo, @Aitozi, the warehoused path is required to determine which FileIO to use. This interface makes sense to me.

Xuanwo · 2024-08-30T12:35:05Z

crates/paimon/src/catalog/mod.rs

+    ) -> Result<()>;
+
+    /// Returns a Table instance for the specified identifier.
+    async fn get_table(&self, identifier: &Identifier) -> Result<impl Table>;


Implementing Table makes this trait non-object safe. Perhaps we could return a Table struct instead.

Implementing Table makes this trait non-object safe. Perhaps we could return a Table struct instead.

Ok, I will create a Table struct instead of trait.

In the java version，Table interface represents various specialized table types. Is it possible to define a simple Table struct in rust ?
What about returning Box<dyn Table> instead?

In the java version，Table interface represents various specialized table types. Is it possible to define a simple Table struct in rust ? What about returning Box<dyn Table> instead?

Maybe we will not use dyn, when returning Box<dyn Table> has some error like error[E0404]: expected trait, found struct Table. In Rust, dyn is used for dynamic dispatch with trait objects.

In the java version，Table interface represents various specialized table types. Is it possible to define a simple Table struct in rust ? What about returning Box<dyn Table> instead?

Maybe we will not use dyn, when returning Box<dyn Table> has some error like error[E0404]: expected trait, found struct Table. In Rust, dyn is used for dynamic dispatch with trait objects.

Why Box<dyn Table> would give error[E0404] ? It should not when using properly.
I think in many cases, the return value of get_table needs to be downcast to a concrete type, so it makes sense to use dyn Table.
Alternatively, maybe using enum Table is feasible

Xuanwo · 2024-08-30T12:36:18Z

crates/paimon/src/catalog/mod.rs

+    async fn list_tables(&self, database_name: &str) -> Result<Vec<String>>;
+
+    /// Checks if a table exists.
+    async fn table_exists(&self, identifier: &Identifier) -> Result<bool> {


It's a bit surprising that we use Identifier in some places but not in others. Shouldn't we be consistent?

It's a bit surprising that we use Identifier in some places but not in others. Shouldn't we be consistent?

In the Java version, Identifier is usually used for table-related methods.

Xuanwo · 2024-08-30T12:37:21Z

crates/paimon/src/catalog/mod.rs

+    ) -> Result<()>;
+
+    /// Returns whether this catalog is case-sensitive.
+    fn case_sensitive(&self) -> bool {


I believe we can consolidate those metadata-related items into a single API.

Xuanwo · 2024-08-30T12:40:29Z

crates/paimon/src/catalog/mod.rs

+/// A table provides basic abstraction for a table type and table scan, and table read.
+///
+/// Impl Reference: <https://github.com/apache/paimon/blob/release-0.8.2/paimon-core/src/main/java/org/apache/paimon/table/Table.java#L41>
+pub trait Table {


I feel like Table should be a struct instead of a trait.

I will create a simple Table struct as the return value of the get_table method. With more design details about Table we can create a new issue for discussion. What do you think of it?

Xuanwo · 2024-08-30T12:41:09Z

Also invite @Aitozi for a review.

XuQianJin-Stars · 2024-08-31T07:44:17Z

crates/paimon/src/catalog/mod.rs

+
+    /// Returns the warehouse root path containing all database directories in this catalog.
+    fn warehouse(&self) -> &str;
+


Do APIs like lockFactory need to be added?

Do APIs like lockFactory need to be added?

I think this PR will add the struct and definition first, maybe we can add some APIs like lockFactory in the further PRs.

JiaQiTang98 · 2025-03-12T13:09:29Z

Hi teams, any update about the PR? Can we merge it and do something based on it?

Xuanwo · 2025-03-12T13:51:09Z

Hi teams, any update about the PR? Can we merge it and do something based on it?

I believe there some unresolved comments left: #62 (comment)

JiaQiTang98 · 2025-03-13T02:17:14Z

@QuakeWang Hi, do you mind if I continue your job?

QuakeWang · 2025-03-18T16:13:09Z

@QuakeWang Hi, do you mind if I continue your job?

Sorry I've been busy recently and didn't check back sooner. Please feel free to do this with this as planned.

feat: add catalog API

f9f957a

Xuanwo reviewed Aug 30, 2024

View reviewed changes

XuQianJin-Stars reviewed Aug 31, 2024

View reviewed changes

SteNicholas changed the title ~~feat: add catalog API~~ feat(catalog): add catalog API Sep 9, 2024

Xuanwo mentioned this pull request Apr 20, 2025

Query for any progress #80

Open

Aitozi mentioned this pull request Sep 6, 2024

Tracking issues of 0.1.0 version for Apache Paimon Rust #3

Open

12 tasks

camilesing mentioned this pull request Apr 25, 2025

feat: add catalog API #85

Open

QuakeWang added 2 commits May 22, 2025 14:11

Merge remote-tracking branch 'upstream/main' into add-catalog

3a92ed3

refactor: convert Table to struct and extract TableOperations

bd0968b


		/// Returns the warehouse root path containing all database directories in this catalog.
		fn warehouse(&self) -> &str;

Uh oh!

feat(catalog): add catalog API #62

Are you sure you want to change the base?

feat(catalog): add catalog API #62

Uh oh!

Conversation

QuakeWang commented Aug 22, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Xuanwo commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JiaQiTang98 commented Mar 12, 2025

Uh oh!

Xuanwo commented Mar 12, 2025

Uh oh!

JiaQiTang98 commented Mar 13, 2025

Uh oh!

QuakeWang commented Mar 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Xuanwo commented Aug 30, 2024 •

edited

Loading