diff --git a/content/v1.10.x/connectors/pipeline/ssis/yaml.md b/content/v1.10.x/connectors/pipeline/ssis/yaml.md index 1272161fc..fc49d750c 100644 --- a/content/v1.10.x/connectors/pipeline/ssis/yaml.md +++ b/content/v1.10.x/connectors/pipeline/ssis/yaml.md @@ -64,17 +64,75 @@ This is a sample config for SSIS: {% /codeInfo %} -{% codeInfo srNumber=1 %} +{% codeInfo srNumber=6 %} -**connection**: +**databaseConnection**: -In terms of `connection` we support the following selections: +In terms of `databaseConnection` we support the following selections: - `Mssql`: Pass the required credentials to reach out this service. We will create a connection to the pointed database and read SSIS data from there. {% /codeInfo %} +{% codeInfo srNumber=7 %} + +**packageConnection**: + +To fetch task dependencies and lineage information from your SSIS pipelines, the connector requires access to the SSIS package XML files. You have two options: + +- **Local Path**: Specify the local directory path where your SSIS package files are stored. The connector will read the XML files directly from this location during metadata extraction. + +{% note noteType="Warning" %} +**Important:** +If you are using the **Local Path** option to provide your SSIS package files, you must run the ingestion workflow through the **CLI** instead of the UI. This is because the ingestion process needs direct access to your local filesystem, which is not available when running ingestion jobs from the UI or server. +{% /note %} + +When configuring the SSIS connector to extract metadata from a **local path**, you need to provide the path up to the directory containing your SSIS project folders. + +For example, if your projects are organized as: +``` +/home/user/repos/ + project1/ + project1/ + ... .dtsx files + project2/ + project2/ + ... .dtsx files +``` + +You should specify the path up to `/home/user/repos/` in your connector configuration. The connector will recursively scan this directory to locate all SSIS project folders and their package XML files. + +- **S3 Bucket**: Upload your SSIS project folders containing the package XML files to an S3 bucket. Then, provide the S3 credentials (such as `awsAccessKeyId`, `awsSecretAccessKey`, and region) along with the bucket name in the connector configuration. The connector will retrieve the package files from your S3 storage. + +When configuring the SSIS connector to extract metadata from S3 storage, you need to upload your SSIS project folders containing all your package files to your S3 bucket. + +Typically, SSIS organizes projects in a structure like: +``` +repos/ + project1/ + project1/ + ... .dtsx files + project2/ + project2/ + ... .dtsx files +``` + +You should upload the inner project folders (e.g., `project1/project1/`, `project2/project2/`, etc.) into your S3 bucket. For example, if your bucket name is `packages`, the structure in your bucket should look like: +``` +packages/ + project1/ + project1/ + ... .dtsx files + project2/ + project2/ + ... .dtsx files +``` + +It is necessary to provide the same bucket name (e.g., `packages`) along with the credentials for your S3 storage when configuring the connector. + +{% /codeInfo %} + {% partial file="/v1.10/connectors/yaml/pipeline/source-config-def.md" /%} {% partial file="/v1.10/connectors/yaml/ingestion-sink-def.md" /%} @@ -101,30 +159,18 @@ source: database: db hostPort: localhost:1433 ``` - -**storage**: -To extract task dependencies and lineage, the connector needs access to your SSIS package XML files. -You can either provide the local path to your project folders, or upload your projects to an S3 bucket and supply the bucket name along with S3 credentials. - -- For s3 - -Pass the `S3` credentials where your package folders are uploaded and specify the bucket name inside the `bucketNames` field. - - ```yaml {% srNumber=7 %} - storageConnection: - type: S3 - awsConfig: - awsAccessKeyId: test - awsSecretAccessKey: test - awsRegion: us-east-2 - endPointURL: https://packages.s3.us-east-2.amazonaws.com - bucketNames: - - bucket_name -``` -- For Local -```yaml {% srNumber=8 %} - localProjectsPath: /home/user/repos/ + packageConnection: /home/user/repos/ + # For S3: + # packageConnection: + # type: S3 + # awsConfig: + # awsAccessKeyId: test + # awsSecretAccessKey: test + # awsRegion: us-east-2 + # endPointURL: https://packages.s3.us-east-2.amazonaws.com + # bucketNames: + # - bucket_name ``` {% partial file="/v1.10/connectors/yaml/pipeline/source-config.md" /%} @@ -138,4 +184,3 @@ Pass the `S3` credentials where your package folders are uploaded and specify th {% /codePreview %} {% partial file="/v1.10/connectors/yaml/ingestion-cli.md" /%} - diff --git a/content/v1.11.x-SNAPSHOT/connectors/pipeline/ssis/yaml.md b/content/v1.11.x-SNAPSHOT/connectors/pipeline/ssis/yaml.md index 7532afd58..95a9df49a 100644 --- a/content/v1.11.x-SNAPSHOT/connectors/pipeline/ssis/yaml.md +++ b/content/v1.11.x-SNAPSHOT/connectors/pipeline/ssis/yaml.md @@ -64,17 +64,75 @@ This is a sample config for SSIS: {% /codeInfo %} -{% codeInfo srNumber=1 %} +{% codeInfo srNumber=6 %} -**connection**: +**databaseConnection**: -In terms of `connection` we support the following selections: +In terms of `databaseConnection` we support the following selections: - `Mssql`: Pass the required credentials to reach out this service. We will create a connection to the pointed database and read SSIS data from there. {% /codeInfo %} +{% codeInfo srNumber=7 %} + +**packageConnection**: + +To fetch task dependencies and lineage information from your SSIS pipelines, the connector requires access to the SSIS package XML files. You have two options: + +- **Local Path**: Specify the local directory path where your SSIS package files are stored. The connector will read the XML files directly from this location during metadata extraction. + +{% note noteType="Warning" %} +**Important:** +If you are using the **Local Path** option to provide your SSIS package files, you must run the ingestion workflow through the **CLI** instead of the UI. This is because the ingestion process needs direct access to your local filesystem, which is not available when running ingestion jobs from the UI or server. +{% /note %} + +When configuring the SSIS connector to extract metadata from a **local path**, you need to provide the path up to the directory containing your SSIS project folders. + +For example, if your projects are organized as: +``` +/home/user/repos/ + project1/ + project1/ + ... .dtsx files + project2/ + project2/ + ... .dtsx files +``` + +You should specify the path up to `/home/user/repos/` in your connector configuration. The connector will recursively scan this directory to locate all SSIS project folders and their package XML files. + +- **S3 Bucket**: Upload your SSIS project folders containing the package XML files to an S3 bucket. Then, provide the S3 credentials (such as `awsAccessKeyId`, `awsSecretAccessKey`, and region) along with the bucket name in the connector configuration. The connector will retrieve the package files from your S3 storage. + +When configuring the SSIS connector to extract metadata from S3 storage, you need to upload your SSIS project folders containing all your package files to your S3 bucket. + +Typically, SSIS organizes projects in a structure like: +``` +repos/ + project1/ + project1/ + ... .dtsx files + project2/ + project2/ + ... .dtsx files +``` + +You should upload the inner project folders (e.g., `project1/project1/`, `project2/project2/`, etc.) into your S3 bucket. For example, if your bucket name is `packages`, the structure in your bucket should look like: +``` +packages/ + project1/ + project1/ + ... .dtsx files + project2/ + project2/ + ... .dtsx files +``` + +It is necessary to provide the same bucket name (e.g., `packages`) along with the credentials for your S3 storage when configuring the connector. + +{% /codeInfo %} + {% partial file="/v1.11/connectors/yaml/pipeline/source-config-def.md" /%} {% partial file="/v1.11/connectors/yaml/ingestion-sink-def.md" /%} @@ -101,30 +159,18 @@ source: database: db hostPort: localhost:1433 ``` - -**storage**: -To extract task dependencies and lineage, the connector needs access to your SSIS package XML files. -You can either provide the local path to your project folders, or upload your projects to an S3 bucket and supply the bucket name along with S3 credentials. - -- For s3 - -Pass the `S3` credentials where your package folders are uploaded and specify the bucket name inside the `bucketNames` field. - - ```yaml {% srNumber=7 %} - storageConnection: - type: S3 - awsConfig: - awsAccessKeyId: test - awsSecretAccessKey: test - awsRegion: us-east-2 - endPointURL: https://packages.s3.us-east-2.amazonaws.com - bucketNames: - - bucket_name -``` -- For Local -```yaml {% srNumber=8 %} - localProjectsPath: /home/user/repos/ + packageConnection: /home/user/repos/ + # For S3: + # packageConnection: + # type: S3 + # awsConfig: + # awsAccessKeyId: test + # awsSecretAccessKey: test + # awsRegion: us-east-2 + # endPointURL: https://packages.s3.us-east-2.amazonaws.com + # bucketNames: + # - bucket_name ``` {% partial file="/v1.11/connectors/yaml/pipeline/source-config.md" /%} @@ -138,4 +184,3 @@ Pass the `S3` credentials where your package folders are uploaded and specify th {% /codePreview %} {% partial file="/v1.11/connectors/yaml/ingestion-cli.md" /%} -