fix: refine doc for write support #999
Conversation
ZENOTME
force-pushed
the
write_example
branch
2 times, most recently
from
f817d1a to
67c1c5a
Compare
ZENOTME
force-pushed
the
write_example
branch
from
67c1c5a to
4b1713d
Compare
|
Feel free for any suggestion to make example better. cc @Xuanwo @liurenjie1024 @Fokko @kevinjqliu @xxchan @jonathanc-n |
| //! | ||
| //! ## Fast append data to table | ||
| //! | ||
| //! ```rust, no_run |
There was a problem hiding this comment.
This test only use memory file io and memory catalog, it should be able to run?
There was a problem hiding this comment.
We should create table here to make it run.
There was a problem hiding this comment.
I personally believe it's more valuable. Is the blocker here that we don't support creating tables yet?
There was a problem hiding this comment.
No, I will add it later.
There was a problem hiding this comment.
I find that the memory catalog can't update now. So we should merge #1002 first.
There was a problem hiding this comment.
Hi, @ZENOTME Thanks for this, but I don't think this is a good example of writing data since it involves too many low level api invocation. We should have a high level api to accept a stream of arrow record batches, and have the transaction, file writer hidden under the hood, just like what we did in table scan to convert table to an arrow batch stream. Or we could integrate with datafusion do this.
There was a problem hiding this comment.
Yes, I hope we can have more high-level api examples. I think this is an example of low-level api which may be useful for computing engines integrated with iceberg-rust.🤔
There was a problem hiding this comment.
This is an example showing in crate front page, which I don't think is appropriate. If we actually want to tell compute engine how to integrate it, a better place would be to place it in website, which appears as an advanced tutorial.
There was a problem hiding this comment.
This is an example showing in crate front page, which I don't think is appropriate.
Looks reasonable to me. What about also place it at https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/writer/mod.rs
| //! .try_into() | ||
| //! .unwrap(), | ||
| //! ); | ||
| //! let location_generator = DefaultLocationGenerator::new(table.metadata().clone()).unwrap(); |
There was a problem hiding this comment.
We are returning Result<()> here, how about using ? here for clean code?
Part of #986. This PR adds more examples to refine the doc of writer.