feat(dbt): enhance DBTCloud integration with bulk job ingestion

[askumar27]

DBT Cloud Auto-Discovery Mode

Overview

This document describes the new auto-discovery mode feature for the DBT Cloud integration in DataHub. This enhancement enables automatic discovery of jobs within a specified DBT Cloud project, eliminating the need to manually specify individual job IDs.

Motivation

Previously, users had to manually specify a single job_id to ingest DBT Cloud metadata. For organizations with multiple jobs in a project, this required creating separate ingestion configurations for each job. Auto-discovery mode solves this by automatically discovering and ingesting all qualifying jobs in a project.

Features

Automatic Job Discovery

• Discovers all jobs for a specified DBT Cloud project
• Filters to only production environment jobs
• Only ingests jobs with generate_docs=True enabled
• Supports optional regex-based job filtering

Dual Mode Operation

The integration now supports two modes:

1. Explicit Mode (existing, backward compatible):

   • Manually specify a single job_id
   • Optionally specify run_id (defaults to latest)

2. Auto-Discovery Mode (new):

   • Automatically discovers jobs for a project
   • Always uses the latest run for each job
   • Filters by production environment and generate_docs flag

Configuration

Basic Auto-Discovery Configuration

source:
  type: "dbt-cloud"
  config:
    # Required fields
    account_id: 107298
    project_id: 466863
    target_platform: "snowflake"
    token: "${DBT_CLOUD_TOKEN}"

    # Auto-discovery configuration
    auto_discovery:
      enabled: true

Advanced Configuration with Job Filtering

source:
  type: "dbt-cloud"
  config:
    account_id: 107298
    project_id: 466863
    target_platform: "snowflake"
    token: "${DBT_CLOUD_TOKEN}"

    # Optional: platform_instance for DBT project identification
    platform_instance: "dbt_cloud_project_466863"

    # Optional: target_platform_instance to link to specific platform instance
    target_platform_instance: "snowflake_prod"

    # Auto-discovery with job filtering
    auto_discovery:
      enabled: true
      job_id_pattern:
        allow:
          - "96.*"      # Only jobs starting with 96
          - "148094"    # Specific job
        deny:
          - ".*test.*"  # Exclude test jobs

Explicit Mode (Backward Compatible)

source:
  type: "dbt-cloud"
  config:
    account_id: 107298
    project_id: 466863
    target_platform: "snowflake"
    token: "${DBT_CLOUD_TOKEN}"

    # Explicit job specification
    job_id: 148094
    run_id: 12345  # Optional, defaults to latest run

Future Enhancements

The AutoDiscoveryConfig structure is designed to support future enhancements:

• Account-level discovery: Discover projects and jobs across an entire account
• Environment filtering: Support non-production environments
• Job status filtering: Filter by job success/failure status
• Schedule-based filtering: Filter jobs by schedule frequency
• Tag-based filtering: Filter jobs by DBT Cloud tags

References

• DBT Cloud API Documentation
• DBT Cloud Discovery API
• DataHub DBT Integration

[codecov]

Codecov Report

❌ Patch coverage is 99.28058% with 1 line in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...tion/src/datahub/ingestion/source/dbt/dbt_cloud.py	99.11%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[alwaysmeticulous]

🔴 Meticulous spotted visual differences in 2 of 1067 screens tested: view and approve differences detected.

Meticulous evaluated ~8 hours of user flows against your PR.

_{Last updated for commit aa64b17. This comment will update as new commits are pushed.}

[sgomezvillamor]

Loved the PR description 🔝

[sgomezvillamor]

Overall LGTM

My main comments/suggestions:

• give more visiblity in the UI to some info/warn/error log traces
• is there anything preventing to make auto-discovery enabled by default?

[aikido-pr-checks]

Potential user input in HTTP request may allow SSRF attack - medium severity
If an attacker can control the URL input leading into this HTTP request, the attack might be able to perform an SSRF attack. This kind of attack is even more dangerous if the application returns the response of the request to the user. It could allow them to retrieve information from higher privileged services within the network (such as the metadata service, which is commonly available in cloud services, and could allow them to retrieve credentials).

Show Remediation

Remediation - medium confidence
This patch mitigates the opening of potentially unsafe URLs by implementing validation for URLs passed to urllib_urlopen.

Suggested change

	url,
	(
	url
	if url.startswith(("https://", "http://"))
	else ValueError("Invalid URL scheme")
	),

^{View details in Aikido Security}

[aikido-pr-checks]

Dangerous use of assert - low severity
When running Python in production in optimized mode, assert calls are not executed. This mode is enabled by setting the PYTHONOPTIMIZE command line flag. Optimized mode is usually ON in production. Any safety check done using assert will not be executed.

Remediation: Raise an exception instead of using assert.
^{View details in Aikido Security}

[codecov]

Bundle Report

Changes will increase total bundle size by 5.22kB (0.02%) ⬆️. This is within the configured threshold ✅

Detailed changes

Bundle name	Size	Change
datahub-react-web-esm	28.64MB	5.22kB (0.02%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: datahub-react-web-esm

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
assets/index-*.js	5.22kB	19.01MB	0.03%

[aikido-pr-checks]

Dangerous use of assert - low severity
When running Python in production in optimized mode, assert calls are not executed. This mode is enabled by setting the PYTHONOPTIMIZE command line flag. Optimized mode is usually ON in production. Any safety check done using assert will not be executed.

Remediation: Raise an exception instead of using assert.
^{View details in Aikido Security}