feat(dbt): add filtering for materialized nodes based on their physical location

[abdullahtariqq]

Overview

Adds advanced filtering for dbt nodes based on their materialized database location to enable catalog consistency across multi-team dbt projects.

Changes

• Added MaterializedNodePatternConfig with database_pattern, schema_pattern, table_pattern fields
• Clean separation: node_name_pattern (internal dbt refs) vs source_pattern (external table location)
• Hierarchical filtering: database → schema → table

Use Case

Problem: Multi-database dbt projects (operational_db.*, analytics_db.*, ...) need consistent catalog filtering
Solution: Match source system ingestion patterns for clean, focused data catalog

Example:

  materialized_node_pattern:
    database_pattern:
      allow: ["analytics_db"]
    schema_pattern:
      allow: ["analytics_db\\.marts.*"]
    table_pattern:
      allow: ["fct_.*", "stg_.*"]
      deny: ["int_.*", "tmp_.*"]

Benefits

• Multi-team separation by database
• Catalog consistency with source ingestion
• Backward compatible - existing configs unchanged
• Performance optimized with early-exit logic

Enables fine-grained dbt ingestion control for complex organizational data architectures.

[codecov]

Bundle Report

Changes will increase total bundle size by 1.43kB (0.0%) ⬆️. This is within the configured threshold ✅

Detailed changes

Bundle name	Size	Change
datahub-react-web-esm	28.57MB	1.43kB (0.0%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: datahub-react-web-esm

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
assets/index-*.js	1.43kB	18.91MB	0.01%

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[skrydal]

Hello @abdullahtariqq , thank you for sharing this contribution. I think the feature is a bit complex, due to the details of dbt references handling. To be specific, node.dbt_name is completely independent of node.schema, node.database or node.name - as the former is an internal dbt reference, while others might refer to an external table.
Therefore, I would introduce a clear separation between those two checks, to avoid confusion, for example we could have separate config field: source_pattern, it should contain trifecta "table", "schema", "database", and follow DataHub-wide pattern matching:

1. database pattern is matched against database name
2. schema pattern is matched against f"{database}.{schema}" (fully qualified name)
3. table pattern is matched against f"{database}.{schema}.{table}" (fully qualified name)

There is an open question whether it should be done against nodes of type source or all external references, what do you think about it?

The _is_node_allowed function would work the following way:

1. if dbt_name is not allowed, return False
2. if node is of type source, then match node.name, node.schema, node.database against the patterns, as specified above

Is this aligned with the use-case you had in mind when creating it?

Of course golden files need to be aligned properly.

[abdullahtariqq]

@skrydal Thank you for the thoughtful feedback! You're absolutely right about the architectural separation.

However, I'd like to clarify the actual use case to ensure we design the right solution:

The filtering need is for materialized nodes (tables/views) rather than just sources. Here's the scenario:

• Multiple teams work within the same dbt project across different databases/schemas
• Team A: operational_db.orders.*, operational_db.inventory.*
• Team B: analytics_db.marts.*, analytics_db.reporting.*
• Some databases contain operational data, others analytical data

The goal is catalog consistency:

• BigQuery/Snowflake/Redshift or other source ingestion filters to analytics_db.* only
• dbt ingestion should filter to the same analytics_db.* to maintain a clean, consistent catalog view
• This prevents operational tables from appearing in the data catalog when they're not meant for analytical consumption

So the filtering should apply to all materialized nodes (table, view, etc.) based on where they're physically materialized, not just sources.

Proposed Configuration Example

  source_pattern:
    database_pattern:
      allow: ["analytics_db", "marts_db"]
    schema_pattern:
      allow: ["analytics_db\\.marts.*", "analytics_db\\.reporting.*"]
    table_pattern:
      allow: ["analytics_db\\.marts\\.customer_.*"]

Benefits

• Internal vs external reference separation
• Matches source system ingestion patterns
• Legacy patterns still supported
• Database-level separation for multi-team projects

Does this align with your architectural vision? I'm happy to adjust the approach based on any additional feedback!

[abdullahtariqq]

@skrydal thank you for the feedback. materialized_nodes_pattern is a good alternative and conveys proper meaning too.

[abdullahtariqq]

@skrydal Thank you for the thorough review. Could you please approve it in case everything looks good?

[skrydal]

Thank you for your contribution to the project.

[skrydal]

FYI @abdullahtariqq this change has been released in acryl-datahub version 1.2.0.10rc2.