Skip to content

Conversation

@btkcodedev
Copy link

Closes #14733

Summary

Adds an HBase metadata ingestion plugin to DataHub to extract namespaces, tables, column families and schema metadata via the HBase Thrift API. Includes backend source implementation, plugin registration, UI config stubs, docs and minimal wiring to surface the connector in the ingestion UI.

Motivation

Support for HBase metadata ingestion so users can index HBase namespaces/tables into DataHub and make them discoverable, searchable and lineage-able.

What changed

Added backend source implementation
Registered plugin entrypoint

The connector is a StatefulIngestionSourceBase and supports:
Containers: HBase namespaces
Schema metadata: row key + column families / qualifiers
Deletion detection via stateful ingestion
Platform instance and env config
Thrift-based connection (Thrift / happybase recommended)

Compatibility / Risk

No breaking changes.
Requires Thrift-related dependencies and (optionally) UI assets.
If Thrift libs missing, connector logs a clear error and fails to connect.

@github-actions github-actions bot added ingestion PR or Issue related to the ingestion of metadata product PR or Issue related to the DataHub UI/UX community-contribution PR or Issue raised by member(s) of DataHub Community labels Nov 10, 2025
@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Nov 10, 2025
@codecov
Copy link

codecov bot commented Nov 10, 2025

Codecov Report

❌ Patch coverage is 7.69231% with 24 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...eb-react/src/app/ingest/source/conf/hbase/hbase.ts 7.69% 12 Missing ⚠️
...-react/src/app/ingestV2/source/conf/hbase/hbase.ts 7.69% 12 Missing ⚠️

📢 Thoughts on this report? Let us know!

@codecov
Copy link

codecov bot commented Nov 10, 2025

Bundle Report

Bundle size has no change ✅

"glue = datahub.ingestion.source.aws.glue:GlueSource",
"sagemaker = datahub.ingestion.source.aws.sagemaker:SagemakerSource",
"hana = datahub.ingestion.source.sql.hana:HanaSource",
"hbase = datahub.ingestion.source.sql.hbase:HBaseSource",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added ✅

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this rectangle shape may not fit well in the UI, have you checked?
a logo that fits better in a square/circle shape would look better

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resized to square

@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Nov 12, 2025
Copy link
Contributor

@sgomezvillamor sgomezvillamor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good

I miss:

@sgomezvillamor
Copy link
Contributor

Please, double check you followed steps defined here https://docs.datahub.com/docs/metadata-ingestion/adding-source Eg I miss updates in constant.ts

@datahub-cyborg datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Nov 13, 2025
@btkcodedev
Copy link
Author

Changes Made:

Backend:

  • Added dependencies (happybase, thrift) to setup.py
  • Fixed linting issues in hbase.py

Frontend:

  • Added HBase to the sources registry datahub-web-react/src/app/ingest/source/builder/sources.json with the default recipe
  • Created UI form fields datahub-web-react/src/app/ingestV2/source/builder/RecipeForm/hbase.ts
  • Updated constants for logo and URN mapping

Documentation:

  • Resized logo to square
  • Added setup guide and sample recipe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PR or Issue raised by member(s) of DataHub Community ingestion PR or Issue related to the ingestion of metadata needs-review Label for PRs that need review from a maintainer. product PR or Issue related to the DataHub UI/UX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

How can I ingest the metadata of HBase?

2 participants