Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feat] MinIO base setup for Iceberg for local testing #124

Open
shubham19may opened this issue Mar 6, 2025 · 3 comments
Open

[Feat] MinIO base setup for Iceberg for local testing #124

shubham19may opened this issue Mar 6, 2025 · 3 comments
Labels
enhancement New feature or request good second issue after you have solved good first issue java-task

Comments

@shubham19may
Copy link
Contributor

Problem

Currently to test Iceberg writer, it needs S3 + Glue or some other catalog. But not everybody has access to it (College students)

Solution

Create a relational catalog in local Postgres and add data in MinIO. This will create a simple enough stack for people to test on local.

@shubham19may shubham19may added enhancement New feature or request good second issue after you have solved good first issue java-task labels Mar 6, 2025
@mrmagicpotato007
Copy link
Contributor

mrmagicpotato007 commented Mar 6, 2025

hey shubham im already using minio in my local i can give a try. why postgres is needed?

@lalit2001
Copy link

For the catalog what should we use Nessi, Iceberg REST catalog, or hive meta store. or user can select any of these?

@shubham19may
Copy link
Contributor Author

@lalit2001 @mrmagicpotato007

I am thinking to create a basic readme/doc on how to do local setup for iceberg writer. similar to how we have around syncing MongoDB (https://olake.io/docs/getting-started/mongodb) you can also edit/add the doc https://github.com/datazip-inc/olake-docs.

For the local better choice would be a JDBC driver like postgres which can be easily setup using a docker compose.

Feel free to checkout to the PR : #113 and test it.

feel free to point out any bugs/issues in code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good second issue after you have solved good first issue java-task
Projects
None yet
Development

No branches or pull requests

3 participants