Skip to content

Commit 850e618

Browse files
albertotessarottoClaudio Benedetti
andauthored
Metadata Inheritance in Data Catalog (#1952)
* Add Metadata Inheritance in Data Catalog * added images * Update docs/data_catalog/frontend/data_catalog_assets.mdx * feat: add newline * update images * little images fix * little adjustments on copy --------- Co-authored-by: Claudio Benedetti <[email protected]>
1 parent de8a62e commit 850e618

19 files changed

+61
-6
lines changed

docs/data_catalog/frontend/data_catalog_assets.mdx

Lines changed: 61 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -58,9 +58,9 @@ Once the searched asset is found among the search returned elements, click on it
5858

5959
![Details of a table page](./../img/table_details_page.png)
6060

61-
At the top of the page a bredcrumb helps users to easily understand the background structure of Data Catalog navigation patterns. In the example, user has entered the table `orders-details` detail page; from the breadcrumb it is noticeable that table `orders-details` belongs to System of Records `crud-books`.
61+
At the top of the page a bredcrumb helps users to easily understand the background structure of Data Catalog navigation patterns. In the example, user has entered the table `AUTHORS` detail page; from the breadcrumb it is noticeable that table `AUTHORS` belongs to System of Records `ORACLE_AWS_CATALOG`.
6262

63-
In case the user wants to easily move from table `orders-details` to SoR `crud-books` without coming back to the search page to find `crud-books` asset, with a simple click on `crud-books` user can enter its detail page.
63+
In case the user wants to easily move from table `AUTHORS` to SoR `ORACLE_AWS_CATALOG` without coming back to the search page to find `AUTHORS` asset, with a simple click on `ORACLE_AWS_CATALOG` user can enter its detail page.
6464

6565
In case the user wants to go back to the page with the items returned by the search query, a click on the `Assets` button located at the beginning of the breadcrumb will take back to the Assets tab where there is the result of the last search performed.
6666

@@ -84,19 +84,19 @@ In case of a **Virtual SoR** detail page, the provider info is not available, be
8484
### Table detail page
8585

8686
The Table detail page shows the following two tabs:
87-
* **General**, that shows a first **Details** section with asset details about name, its SoR name, number of table columns, tags and description of the asset. Moreover, below the Details, there is the **Custom Properties** section, where user can choose among the available custom properties for performing metadata enrichment. This topic is described in details in the [metadata enrichment](#metadata-enrichment) documentation section.
87+
* **General**, that shows a first **Details** section with asset details about namespace, number of table columns, tags and description of the asset. The namespace identifies the context in which the table is stored. It depends on the [provider](/data_catalog/frontend/data_catalog_connections.mdx#connection-providers). Moreover, below the Details, there is the **Custom Properties** section, where user can choose among the available custom properties for performing metadata enrichment. This topic is described in details in the [metadata enrichment](#metadata-enrichment) documentation section.
8888
* **Columns**, that shows the list of tables that belong to that SoR. By clicking on one element of the list, user enters that table detail page.
8989
* **Lineage**, that shows the lineage canvas displaying how a specific table is related to other tables. For more information about the Table-level lineage, visit the [related documentation](/data_catalog/frontend/data_lineage.mdx#table-level-lineage).
9090

9191
![Details of a table page](./../img/table_details_page.png)
9292

9393
:::info
94-
In case of a **Virtual Table** detail page, the Columns tab it is not present. To know more about the concept of `Virtual Asset`, please give a look at the [related documentation](/data_catalog/frontend/data_lineage.mdx#real-and-virtual-assets)
94+
In case of a **Virtual Table** detail page, the Columns tab it is not present. To know more about the concept of `Virtual Asset`, please give a look at the [related documentation](/data_catalog/frontend/data_lineage.mdx#real-and-virtual-assets).
9595
:::
9696

9797
### Column detail page
9898

99-
The Column detail page shows a first **Details** section with asset details such as name, its table name, tags, description of the asset, and many other retrieved info. Moreover, below the Details, there is the **Custom Properties** section, where user can choose among the available custom properties for performing metadata enrichment. This topic is described in details in the [metadata enrichment](#metadata-enrichment) documentation section.
99+
The Column detail page shows a first **Details** section with asset details such as namespace, tags, description of the asset, and many other retrieved info. Moreover, below the Details, there is the **Custom Properties** section, where user can choose among the available custom properties for performing metadata enrichment. This topic is described in details in the [metadata enrichment](#metadata-enrichment) documentation section.
100100

101101
![Details of a column page](./../img/column_details_page.png)
102102

@@ -238,12 +238,67 @@ In case the unknown metadata is no more consistent or useful on the specific ass
238238
Please note that this action leads to the removal of the metadata value saved on DB for that specific asset. The custom property remains instead applied on all the other assets that match the expected asset type.
239239
:::
240240

241+
### Metadata Inheritance
242+
243+
:::caution
244+
This feature is currently in **BETA**, so pay attention when using it.
245+
246+
This feature is supported by versions of [Fabric BFF](/data_catalog/data_catalog_fabric_bff.mdx), [Open Lineage](/data_catalog/data_catalog_open_lineage.mdx), [Data Catalog Frontend](/data_catalog/frontend/overview.mdx) >= v0.3.x. Give a look at the [Data Catalog service latest versions](/data_catalog/compatibility_matrix.md#service-latest-versions).
247+
:::
248+
249+
The **Metadata Inheritance** feature allows the propagation of metadata from parent assets to their child assets.
250+
This ensures consistency and simplifies data governance.
251+
The inheritance mechanism follows the hierarchical structure in Data Catalog: **SoR (System of Record)****Tables****Columns**.
252+
253+
:::info
254+
Metadata Inheritance feature is applicable **exclusively on [real assets](/data_catalog/frontend/data_lineage.mdx#assets-real-and-virtual)**.
255+
:::
256+
257+
#### Applying Custom Properties with Inheritance
258+
259+
When adding a custom property to a **SoR** or a **Table**, a flag can be enabled to propagate the metadata to all its child assets:
260+
261+
<div style={{display: 'flex', justifyContent: 'center'}}>
262+
<div style={{display: 'flex', width: '400px'}}>
263+
264+
![Add custom property with inheritance](./../img/add_custom_property_with_inheritance.png)
265+
266+
</div>
267+
</div>
268+
269+
- If a custom property is added to a **SoR**, enabling inheritance will apply the property to all **Tables** and **Columns** under that SoR.
270+
- If a custom property is added to a **Table**, enabling inheritance will apply the property to all its **Columns**.
271+
272+
Inherited properties will be visible at both **Table** and **Column** levels (or only at **Column** level, this depends on where in the hierarchy level the inheritance has been applied), marked with an ***Inherited*** tag indicating their origin (either from a SoR or a Table).
273+
274+
![Inherited custom property](./../img/inherited_custom_property.png)
275+
276+
#### Updating and Deleting Inherited Properties
277+
278+
- **Updates**: Any update to an inherited custom property at the parent level (SoR or Table) will **overwrite** the value across all child assets (if already present).
279+
- **Deletions**: Removing a custom property at the parent level will also remove it from all child assets.
280+
281+
:::info
282+
Child assets (Tables and/or Columns) that inherit properties **cannot** modify or delete them specifically. The inheritance is strictly enforced from the parent level.
283+
:::
284+
285+
#### Example Use Case
286+
287+
1. A custom property **"Location"** is added to **SoR X** with inheritance enabled.
288+
2. All **Tables** and **Columns** belonging to **SoR X** automatically receive the **"Location"** property with the assigned value.
289+
3. If the **"Location"** property is updated at **SoR X**, the new value is applied to all child **Tables** and **Columns**.
290+
4. If the **"Location"** property is deleted at **SoR X**, it is also removed from all child **Tables** and **Columns**.
291+
5. A user accessing a **Column** under **SoR X** can see the inherited property but cannot modify or remove it.
292+
293+
In case the inherited custom property is displayed among the Unknown Custom Properties card, you can go to the [Custom Properties Management tab](#custom-properties-management) and modify consistently the *Applicable asset type* field for that specific custom property.
294+
241295
## Bulk Actions
242296

243297
Data Catalog UI allows to perform bulk actions on metadata enrichment.
244298
From the search list results, user can decide to select one or more assets; then click the `Bulk edit` button.
245299

246-
A modal opens showing the number and type of assets selected (SoR, table, column). Then the user can choose which fields are to be bulk edited: you can select the asset details description, tags, or choose a list of custom properties that are consistent with the type of assets that have been selected for bulk edit.
300+
A modal opens showing the number and type of assets selected (SoR, table, column).
301+
Then the user can choose which fields are to be bulk edited: you can select the asset details description, tags, or choose a list of custom properties that are consistent with the type of assets that have been selected for bulk edit.
247302

248303
<div style={{display: 'flex', justifyContent: 'center'}}>
249304
<div style={{display: 'flex', width: '700px'}}>
Loading
-58.6 KB
Loading
Loading
-14.8 KB
Loading
-39.2 KB
Loading
35.1 KB
Loading
-23.7 KB
Loading
-124 KB
Loading
Loading
Loading
-48.8 KB
Loading
18.2 KB
Loading
-8.92 KB
Loading
39.8 KB
Loading

docs/data_catalog/img/sor-lineage.png

14.4 KB
Loading
-15.2 KB
Loading
-12.2 KB
Loading
28.9 KB
Loading

0 commit comments

Comments
 (0)