Skip to content

Docs: SSIS YAML Updation #566

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Docs: SSIS YAML Updation #566

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 72 additions & 27 deletions content/v1.10.x/connectors/pipeline/ssis/yaml.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,17 +64,75 @@ This is a sample config for SSIS:

{% /codeInfo %}

{% codeInfo srNumber=1 %}
{% codeInfo srNumber=6 %}

**connection**:
**databaseConnection**:

In terms of `connection` we support the following selections:
In terms of `databaseConnection` we support the following selections:

- `Mssql`: Pass the required credentials to reach out this service. We will
create a connection to the pointed database and read SSIS data from there.

{% /codeInfo %}

{% codeInfo srNumber=7 %}

**packageConnection**:

To fetch task dependencies and lineage information from your SSIS pipelines, the connector requires access to the SSIS package XML files. You have two options:

- **Local Path**: Specify the local directory path where your SSIS package files are stored. The connector will read the XML files directly from this location during metadata extraction.

{% note noteType="Warning" %}
**Important:**
If you are using the **Local Path** option to provide your SSIS package files, you must run the ingestion workflow through the **CLI** instead of the UI. This is because the ingestion process needs direct access to your local filesystem, which is not available when running ingestion jobs from the UI or server.
{% /note %}

When configuring the SSIS connector to extract metadata from a **local path**, you need to provide the path up to the directory containing your SSIS project folders.

For example, if your projects are organized as:
```
/home/user/repos/
project1/
project1/
... .dtsx files
project2/
project2/
... .dtsx files
```

You should specify the path up to `/home/user/repos/` in your connector configuration. The connector will recursively scan this directory to locate all SSIS project folders and their package XML files.

- **S3 Bucket**: Upload your SSIS project folders containing the package XML files to an S3 bucket. Then, provide the S3 credentials (such as `awsAccessKeyId`, `awsSecretAccessKey`, and region) along with the bucket name in the connector configuration. The connector will retrieve the package files from your S3 storage.

When configuring the SSIS connector to extract metadata from S3 storage, you need to upload your SSIS project folders containing all your package files to your S3 bucket.

Typically, SSIS organizes projects in a structure like:
```
repos/
project1/
project1/
... .dtsx files
project2/
project2/
... .dtsx files
```

You should upload the inner project folders (e.g., `project1/project1/`, `project2/project2/`, etc.) into your S3 bucket. For example, if your bucket name is `packages`, the structure in your bucket should look like:
```
packages/
project1/
project1/
... .dtsx files
project2/
project2/
... .dtsx files
```

It is necessary to provide the same bucket name (e.g., `packages`) along with the credentials for your S3 storage when configuring the connector.

{% /codeInfo %}

{% partial file="/v1.10/connectors/yaml/pipeline/source-config-def.md" /%}

{% partial file="/v1.10/connectors/yaml/ingestion-sink-def.md" /%}
Expand All @@ -101,30 +159,18 @@ source:
database: db
hostPort: localhost:1433
```

**storage**:
To extract task dependencies and lineage, the connector needs access to your SSIS package XML files.
You can either provide the local path to your project folders, or upload your projects to an S3 bucket and supply the bucket name along with S3 credentials.

- For s3

Pass the `S3` credentials where your package folders are uploaded and specify the bucket name inside the `bucketNames` field.


```yaml {% srNumber=7 %}
storageConnection:
type: S3
awsConfig:
awsAccessKeyId: test
awsSecretAccessKey: test
awsRegion: us-east-2
endPointURL: https://packages.s3.us-east-2.amazonaws.com
bucketNames:
- bucket_name
```
- For Local
```yaml {% srNumber=8 %}
localProjectsPath: /home/user/repos/
packageConnection: /home/user/repos/
# For S3:
# packageConnection:
# type: S3
# awsConfig:
# awsAccessKeyId: test
# awsSecretAccessKey: test
# awsRegion: us-east-2
# endPointURL: https://packages.s3.us-east-2.amazonaws.com
# bucketNames:
# - bucket_name
```

{% partial file="/v1.10/connectors/yaml/pipeline/source-config.md" /%}
Expand All @@ -138,4 +184,3 @@ Pass the `S3` credentials where your package folders are uploaded and specify th
{% /codePreview %}

{% partial file="/v1.10/connectors/yaml/ingestion-cli.md" /%}

99 changes: 72 additions & 27 deletions content/v1.11.x-SNAPSHOT/connectors/pipeline/ssis/yaml.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,17 +64,75 @@ This is a sample config for SSIS:

{% /codeInfo %}

{% codeInfo srNumber=1 %}
{% codeInfo srNumber=6 %}

**connection**:
**databaseConnection**:

In terms of `connection` we support the following selections:
In terms of `databaseConnection` we support the following selections:

- `Mssql`: Pass the required credentials to reach out this service. We will
create a connection to the pointed database and read SSIS data from there.

{% /codeInfo %}

{% codeInfo srNumber=7 %}

**packageConnection**:

To fetch task dependencies and lineage information from your SSIS pipelines, the connector requires access to the SSIS package XML files. You have two options:

- **Local Path**: Specify the local directory path where your SSIS package files are stored. The connector will read the XML files directly from this location during metadata extraction.

{% note noteType="Warning" %}
**Important:**
If you are using the **Local Path** option to provide your SSIS package files, you must run the ingestion workflow through the **CLI** instead of the UI. This is because the ingestion process needs direct access to your local filesystem, which is not available when running ingestion jobs from the UI or server.
{% /note %}

When configuring the SSIS connector to extract metadata from a **local path**, you need to provide the path up to the directory containing your SSIS project folders.

For example, if your projects are organized as:
```
/home/user/repos/
project1/
project1/
... .dtsx files
project2/
project2/
... .dtsx files
```

You should specify the path up to `/home/user/repos/` in your connector configuration. The connector will recursively scan this directory to locate all SSIS project folders and their package XML files.

- **S3 Bucket**: Upload your SSIS project folders containing the package XML files to an S3 bucket. Then, provide the S3 credentials (such as `awsAccessKeyId`, `awsSecretAccessKey`, and region) along with the bucket name in the connector configuration. The connector will retrieve the package files from your S3 storage.

When configuring the SSIS connector to extract metadata from S3 storage, you need to upload your SSIS project folders containing all your package files to your S3 bucket.

Typically, SSIS organizes projects in a structure like:
```
repos/
project1/
project1/
... .dtsx files
project2/
project2/
... .dtsx files
```

You should upload the inner project folders (e.g., `project1/project1/`, `project2/project2/`, etc.) into your S3 bucket. For example, if your bucket name is `packages`, the structure in your bucket should look like:
```
packages/
project1/
project1/
... .dtsx files
project2/
project2/
... .dtsx files
```

It is necessary to provide the same bucket name (e.g., `packages`) along with the credentials for your S3 storage when configuring the connector.

{% /codeInfo %}

{% partial file="/v1.11/connectors/yaml/pipeline/source-config-def.md" /%}

{% partial file="/v1.11/connectors/yaml/ingestion-sink-def.md" /%}
Expand All @@ -101,30 +159,18 @@ source:
database: db
hostPort: localhost:1433
```

**storage**:
To extract task dependencies and lineage, the connector needs access to your SSIS package XML files.
You can either provide the local path to your project folders, or upload your projects to an S3 bucket and supply the bucket name along with S3 credentials.

- For s3

Pass the `S3` credentials where your package folders are uploaded and specify the bucket name inside the `bucketNames` field.


```yaml {% srNumber=7 %}
storageConnection:
type: S3
awsConfig:
awsAccessKeyId: test
awsSecretAccessKey: test
awsRegion: us-east-2
endPointURL: https://packages.s3.us-east-2.amazonaws.com
bucketNames:
- bucket_name
```
- For Local
```yaml {% srNumber=8 %}
localProjectsPath: /home/user/repos/
packageConnection: /home/user/repos/
# For S3:
# packageConnection:
# type: S3
# awsConfig:
# awsAccessKeyId: test
# awsSecretAccessKey: test
# awsRegion: us-east-2
# endPointURL: https://packages.s3.us-east-2.amazonaws.com
# bucketNames:
# - bucket_name
```

{% partial file="/v1.11/connectors/yaml/pipeline/source-config.md" /%}
Expand All @@ -138,4 +184,3 @@ Pass the `S3` credentials where your package folders are uploaded and specify th
{% /codePreview %}

{% partial file="/v1.11/connectors/yaml/ingestion-cli.md" /%}