Skip to content

Files

Latest commit

389a404 · Feb 22, 2025

History

History

metadata-ingestion

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
Feb 10, 2025
Feb 18, 2025
Aug 28, 2024
Aug 30, 2023
Jan 24, 2025
Dec 16, 2024
Feb 22, 2025
Feb 22, 2025
Feb 16, 2021
Jan 24, 2025
Jan 23, 2025
Jul 25, 2024
Oct 4, 2024
Mar 18, 2024
Feb 20, 2025
Jul 30, 2024
Jan 18, 2025
Feb 13, 2025
Jul 26, 2024
Feb 18, 2025
Feb 22, 2025
Jun 11, 2024
Mar 30, 2022
Nov 20, 2023

Introduction to Metadata Ingestion

:::tip Find Integration Source Please see our Integrations page to browse our ingestion sources and filter on their features. :::

Integration Methods

DataHub offers three methods for data ingestion:

  • UI Ingestion : Easily configure and execute a metadata ingestion pipeline through the UI.
  • CLI Ingestion guide : Configure the ingestion pipeline using YAML and execute by it through CLI.
  • SDK-based ingestion : Use Python Emitter or Java emitter to programmatically control the ingestion pipelines.

Types of Integration

Integration can be divided into two concepts based on the method:

Push-based Integration

Push-based integrations allow you to emit metadata directly from your data systems when metadata changes. Examples of push-based integrations include Airflow, Spark, Great Expectations and Protobuf Schemas. This allows you to get low-latency metadata integration from the "active" agents in your data ecosystem.

Pull-based Integration

Pull-based integrations allow you to "crawl" or "ingest" metadata from the data systems by connecting to them and extracting metadata in a batch or incremental-batch manner. Examples of pull-based integrations include BigQuery, Snowflake, Looker, Tableau and many others.

Core Concepts

The following are the core concepts related to ingestion:

  • Sources: Data systems from which extract metadata. (e.g. BigQuery, MySQL)
  • Sinks: Destination for metadata (e.g. File, DataHub)
  • Recipe: The main configuration for ingestion in the form or .yaml file

For more advanced guides, please refer to the following: