Skip to content

Commit 80e746c

Browse files
amountainramflower-of-the-bridgesalbertotessarottodanibix95
authored
Edit/data fabric open lineage migration (#1940)
* add: HEAD routes * fix: console communication bff snippets * edit a doc link * added a note in CPO docs page * removing obsolete note in projections page * Update docs/data_catalog/data_catalog_fabric_bff.mdx Co-authored-by: Alberto Tessarotto <[email protected]> * Update docs/fast_data/runtime_management/control_plane.mdx * wip * feat(fabric): update endpoints and comp matrix for fabric MSs * docs(control-plane): fix link * Update secure_access.mdx * Update control_plane_frontend.mdx with the change of the default behaviour at first deploy * feat(fast-data): add workload resume chapter to runtime_management workloads * add: control-plane migration guide * Update docs/data_catalog/compatibility_matrix.md --------- Co-authored-by: flower-of-the-bridges <[email protected]> Co-authored-by: Alberto Tessarotto <[email protected]> Co-authored-by: Giovanni Fiordeponti <[email protected]> Co-authored-by: Daniele Bissoli <[email protected]>
1 parent 850e618 commit 80e746c

15 files changed

+240
-164
lines changed

docs/data_catalog/compatibility_matrix.md

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -14,17 +14,27 @@ Please ensure that versions shown in the matrix are respected in your deployed e
1414

1515
| Service | Version | MongoDB | Redis |
1616
| ----------------------------------------------------------- | --------------- | ------- | ------ |
17-
| [Fabric BFF](/data_catalog/data_catalog_fabric_bff.mdx) | 0.1.x - 0.2.x | \>=5.0 | _N/A_ |
18-
| [Open Lineage](/data_catalog/data_catalog_open_lineage.mdx) | 0.1.x - 0.2.x | \>=5.0 | \>=7.0 |
19-
| [Job Runner](/data_catalog/data_catalog_job_runner.mdx) | 0.1.x | \>=5.0 | _N/A_ |
17+
| [Fabric BFF](/data_catalog/data_catalog_fabric_bff.mdx) | 0.1.x - 0.3.x | \>=5.0 | _N/A_ |
18+
| [Open Lineage](/data_catalog/data_catalog_open_lineage.mdx) | 0.1.x - 0.3.x | \>=5.0 | \>=7.0 |
19+
| [Job Runner](/data_catalog/data_catalog_job_runner.mdx) | 0.1.x - 0.2.x | \>=5.0 | _N/A_ |
20+
| [Fabric Admin](/data_catalog/database_setup.mdx) | 0.1.x - 0.4.x | \>=5.0 | _N/A_ |
2021
<p><sup>*</sup><em>N/A</em> means the service does not depend on the resource</p>
2122

2223
## Service Latest Versions
2324

2425
| Service | Version |
2526
| ---------------------------------------------------------------- | ------- |
26-
| [Fabric BFF](/data_catalog/data_catalog_fabric_bff.mdx) | 0.2.1 |
27-
| [Open Lineage](/data_catalog/data_catalog_open_lineage.mdx) | 0.2.2 |
28-
| [Job Runner](/data_catalog/data_catalog_job_runner.mdx) | 0.1.0 |
29-
| [Data Catalog Frontend](/data_catalog/frontend/overview.mdx) | 0.2.1 |
30-
| [Fabric Admin](/data_catalog/database_setup.mdx) | 0.2.0 |
27+
| [Fabric BFF](/data_catalog/data_catalog_fabric_bff.mdx) | 0.3.0 |
28+
| [Open Lineage](/data_catalog/data_catalog_open_lineage.mdx) | 0.3.1 |
29+
| [Job Runner](/data_catalog/data_catalog_job_runner.mdx) | 0.2.0 |
30+
| [Data Catalog Frontend](/data_catalog/frontend/overview.mdx) | 0.3.0 |
31+
| [Fabric Admin](/data_catalog/database_setup.mdx) | 0.4.0 |
32+
33+
## Internal Compatibility
34+
35+
| Service | Fabric BFF | Open Lineage | Job Runner |
36+
| ----------------------------------------------------------------: | :-: | :-: | :-: |
37+
| [Fabric BFF](/data_catalog/data_catalog_fabric_bff.mdx) - 0.3.0 | N/A | 0.3.0 | 0.2.0 |
38+
| [Open Lineage](/data_catalog/data_catalog_open_lineage.mdx) - 0.3.0 | 0.3.0 | N/A | 0.2.0 |
39+
| [Job Runner](/data_catalog/data_catalog_job_runner.mdx) - 0.2.0 | 0.3.0 | 0.3.0 | N/A |
40+
| [Data Catalog Frontend](/data_catalog/frontend/overview.mdx) - 0.3.0 | 0.3.0 | N/A | N/A |

docs/data_catalog/data_catalog_fabric_bff.mdx

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -85,19 +85,19 @@ Here can be found an example of configuration that assumes Fabric BFF and Job Ru
8585

8686
#### Console Communication
8787

88-
In order for Data Catalog UI to know which environments can be linked to [Mia Platform CRUD connections](#todo-crud-section),
88+
In order for Data Catalog UI to know which environments can be linked to [Mia Platform CRUD connections](/data_catalog/frontend/data_catalog_connections.mdx#mia-platform-crud),
8989
the service needs to contact Mia-Platform Console and retrieve the list of Projects that should be accessible from this Data Catalog instance.
9090

9191
To achieve so, it is first necessary that your Company Owner creates a dedicated [Service Account](/development_suite/identity-and-access-management/manage-service-accounts.md)
9292
on your Mia-Platform Console instance and assign to it the proper permissions for listing the Console projects of interest.
9393

9494
:::tip Good practices in permissions assignment
9595
Pay attention to the level of access to the resources that you assign to the Service Account.
96-
For Control Plane use case, a good practice may be to assign the role of `guest` at Company level while granting
97-
the `reporter` role to all the projects that should be visible by Control Plane.
96+
For Data Catalog use case, a good practice may be to assign the role of `guest` at Company level while granting
97+
the `reporter` role to all the projects that should be visible by Data Catalog.
9898
About permissions assignment, it is possible to go even more granular in case you want to allow visibility only to a subset of runtime environments of a specific project.
9999
In fact, to do that, you may opt to assign the role of `guest` even at Project level while granting
100-
the `reporter` role solely to those runtime environments that should be visible by Control Plane.
100+
the `reporter` role solely to those runtime environments that should be visible by Data Catalog.
101101
:::
102102

103103
Once the service account has been registered, your Company Owner needs to hand over to you its credentials, which are:
@@ -115,7 +115,7 @@ These details then should be inserted in your Fabric BFF service configuration u
115115
:::caution
116116
It is responsibility of your Company Owner to ensure that service account credentials are properly processed according to your company security policies.
117117

118-
Furthermore, it is of <u>extreme importance</u> understanding that **any** Control Plane user will be able to list the project name
118+
Furthermore, it is of <u>extreme importance</u> understanding that **any** Data Catalog user with enough permissions for [connections management](/data_catalog/frontend/data_catalog_connections.mdx) will be able to view the project name
119119
and available environments of all the projects that can be accessed by the service account configured on Fabric BFF.
120120
:::
121121

@@ -125,10 +125,10 @@ This is and example of `console` property configuration:
125125

126126
:::tip
127127
The following properties support [secret resolution](/fast_data/configuration/secrets_resolution.md):
128-
- `console.target`
129-
- `console.auth.credentials.clientId`
130-
- `console.auth.credentials.clientKeyId`
131-
- `console.auth.credentials.privateKey`
128+
- `console.rest.target`
129+
- `console.rest.auth.credentials.clientId`
130+
- `console.rest.auth.credentials.clientKeyId`
131+
- `console.rest.auth.credentials.privateKey`
132132
:::
133133

134134
A custom x509 certificate can be added to the default root keychain of certificates for any client/reversed-proxy reached by Fabric BFF.
@@ -230,6 +230,7 @@ The following routes are exposed over the `/api/job-runner` endpoint and are for
230230

231231
| Route | Type | Method | Description |
232232
|------------------------------|-----------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
233+
| `/feedback` | Websocket | HEAD | Route used by Websocket for authentication. |
233234
| `/feedback` | Websocket | GET | Route for receiving feedback messages from the `Status` method of the [Job Runner gRPC service.](/data_catalog/data_catalog_job_runner.mdx#job-runner) |
234235
| `/job-runner/*` | REST | * | Routes prefixed with `/job-runner` are converted into gRPC requests towards Job Runner service.<br/> For more details please read [corresponding documentation](/data_catalog/data_catalog_job_runner.mdx#grpc-services)|
235236
| `/agent/drivers` | REST | GET | Route for invoking the `ListDrivers` method of the [ODBC Client gRPC service.](/data_catalog/data_catalog_job_runner.mdx#odbc-client) |

docs/data_catalog/data_catalog_open_lineage.mdx

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,7 @@ The following routes are exposed over the `/api/data-catalog` endpoint.
217217
|------------------------------------------------------------|-----------|--------|-----------------------------------------------------------------------------------------------------------|
218218
| `/assets/search` | REST | GET | Search for dataset assets and their metadata |
219219
| `/assets/search-parents` | REST | GET | Search for name of system of record or table name |
220+
| `/bulk-actions` | Websocket | HEAD | Route used by Websocket for authentication. |
220221
| `/bulk-actions` | Websocket | GET | Route for handling async operations over multiple datasets records |
221222
| `/tags/count` | REST | GET | Count how many unique tags exists among all data assets |
222223
| `/tags/items` | REST | GET | List existing tags associated to data assets |
@@ -279,3 +280,10 @@ The following steps needs to be followed:
279280

280281
See [Routes paragraph](#open-lineage-routes) to more details. Remember also to update the [users management microfrontend](/data_catalog/secure_access.mdx#users-management)
281282
to support the new permission `update:lineage`.
283+
284+
### From 0.2.x to 0.3.0
285+
286+
- A migration CLI must run to update data-fabric records. To do so, you must retrieve the docker image `data-fabric/fabric_admin:0.4.0` and run the following subcommand
287+
```shell
288+
docker run --pull=always <registry-url>/data-fabric/fabric-admin:latest open-lineage update-records --url=<MONGODB_URL>
289+
```

docs/data_catalog/secure_access.mdx

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -599,8 +599,8 @@ all the functionalities of Data Catalog system, both the frontend and backend co
599599
| Endpoint | Service | Authentication Required | User Group Permission |
600600
|---------------------|-----------------|:-----------------------:|-------------------------------------|
601601
| `/api/connections` | fabric-bff | ✅ | `false` |
602-
| `/api/data-catalog` | fabric-bff | ✅ | `permissions["update:bulk-action"]` |
603-
| `/api/job-runner` | fabric-bff | ✅ | `permissions["admin:connections"]` |
602+
| `/api/data-catalog` | fabric-bff | ✅ | `false` |
603+
| `/api/job-runner` | fabric-bff | ✅ | `false` |
604604
| `/api/open-lineage` | fabric-bff | ✅ | `false` |
605605
| `/data-catalog` | data-catalog-fe | ✅ | `true` |
606606

@@ -616,6 +616,9 @@ is listed here:
616616
| `update:metadata-assets` | enable editing metadata associated to data catalog records |
617617
| `update:bulk-action` | allow users performing the same operation in bulk over multiple data catalog records |
618618
| `update:lineage` | allow users to update lineage information of assets |
619+
| `admin:producers` | enable to trigger the execution of different jobs that can update the state of the Data Catalog solution |
620+
| `read:users` | enables a user to access the user management application in read-only mode |
621+
| `update:users` | enables a user to edit users details in the user management application |
619622

620623
:::caution
621624
Please ensure that all these endpoints and subsequent routes are set with _Authentication Required_ in their security details tab.
@@ -670,6 +673,7 @@ so that each operation is covered with the correct grant.
670673
| `/metadata-registry/items/:name` | REST | PATCH | `permissions["update:metadata-assets"]` |
671674
| `/metadata-registry/items/:name` | REST | DELETE | `permissions["update:metadata-assets"]` |
672675
| `/metadata-registry/search` | REST | GET | `permissions["read:data-assets"]` |
676+
| `/bulk-actions` | Websocket | HEAD | `permissions["update:bulk-action"]` |
673677
| `/bulk-actions` | Websocket | GET | `permissions["update:bulk-action"]` |
674678

675679
#### Job Runner API
@@ -685,6 +689,7 @@ so that each operation is covered with the correct grant.
685689
| `/agent/dsn` | REST | GET | `permissions["admin:connections"]` |
686690
| `/agent/drivers` | REST | GET | `permissions["admin:connections"]` |
687691
| `/feedback` | Websocket | GET | `permissions["admin:connections"]` |
692+
| `/feedback` | Websocket | HEAD | `permissions["admin:connections"]` |
688693

689694
#### Open Lineage API
690695

@@ -698,10 +703,10 @@ so that each operation is covered with the correct grant.
698703
| `/jobs/items` | REST | POST | `permissions["update:lineage"]` |
699704
| `/jobs/items/:id` | REST | PATCH | `permissions["update:lineage"]` |
700705
| `/jobs/items/:id` | REST | DELETE | `permissions["update:lineage"]` |
701-
| `/dataset/items/:id` | REST | GET | `permissions["read:data-assets"]` |
702-
| `/dataset/items` | REST | POST | `permissions["update:lineage"]` |
703-
| `/dataset/items/:id` | REST | PATCH | `permissions["update:lineage"]` |
704-
| `/dataset/items/:id` | REST | DELETE | `permissions["update:lineage"]` |
706+
| `/datasets/items/:id` | REST | GET | `permissions["read:data-assets"]` |
707+
| `/datasets/items` | REST | POST | `permissions["update:lineage"]` |
708+
| `/datasets/items/:id` | REST | PATCH | `permissions["update:lineage"]` |
709+
| `/datasets/items/:id` | REST | DELETE | `permissions["update:lineage"]` |
705710
| `/facets/storage/items/:id` | REST | GET | `permissions["read:data-assets"]` |
706711
| `/facets/storage/items/:id` | REST | DELETE | `permissions["update:lineage"]` |
707712

@@ -1368,4 +1373,4 @@ showing a set of user's details and scrolling it will be possible to access the
13681373

13691374
Once the needed permissions are added to the `permissions`, click the _Update Data_ button to store the changes.
13701375
Now, the user who has just received the new permissions **must log out and login again** to be able to access
1371-
Data Catalog UI with the new set of grants.
1376+
Data Catalog UI with the new set of grants.

docs/development_suite/api-console/api-design/endpoints.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,7 +225,7 @@ The **Advanced** tab is visible only if the Envoy API Gateway service is enabled
225225
- **Rate limit** (_integer_): the maximum frequency (in terms of requests per second) with which requests are forwarded to the underlying service;
226226
- **Request body size** (_decimal_): the maximum body size of user requests.
227227
- **Iframe embedding options**: the X-Frame-Options directive that is considered when the endpoint response should be embedded in an iframe;
228-
- **Protcol options**: this options instruct Envoy to process the request with the protocol coming from the downstream connection, allowing to dinamically infer the protocol to be used (HTTP/1.1 or HTTP/2);
228+
- **Protocol options**: this options instruct Envoy to process the request with the protocol coming from the downstream connection, allowing to dinamically infer the protocol to be used (HTTP/1.1 or HTTP/2);
229229

230230
:::warning
231231
This `Iframe embedding` option is configurable only for the `Envoy API Gateway`, instead for `Nginx API Gateway` it is required to configure it manually using the `Advanced` section of the Console

docs/fast_data/configuration/projections.md

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,6 @@ The creation of a System of Record requires you to insert a System ID, which is
1818

1919
The System of Record is then created.
2020

21-
:::note
22-
In case it is not possible to find the button `Create new System of Record`, it means that a project may have been configured
23-
to expose Systems of Record under the [Data Catalog](/data_catalog/overview.mdx) feature, which allows to visualize them in a read-only fashion.
24-
:::
25-
2621
## Delete a System of Record
2722

2823
To delete a System of Record, you have to click the `Delete` button in the bottom-right corner of the System of Record detail page.

0 commit comments

Comments
 (0)