Description

Dalgo behind the hoods relies on airbyte for running sync (moving data) from a source to a destination. We want an integration test suite for this that will help & give us confidence while doing upgrades. The integration test should run an end to end flow to include the following steps

• Create a new airbyte workspace. Everything below should happen in the workspace
• The test suite should create a destination or warehouse (this should be configurable, maybe an .env)
• Create a bunch of sources (custom like kobotoolbox or standard ones like google sheets). The list of sources should also be configurable.
• Create connections between all possible sources & destinations. By default select all streams and set the sync mode to Full refresh overwrite
• Run all connection syncs one by one and wait for them to finish
• For any connection, validate fetching of sync history and logs for them
• Tear down the airbyte workspace & all associated entities.

Goals

Expected Outcome

Acceptance Criteria

• An integration test suite

Implementation Details

The idea would be to use pytest. We already have some part of this implemented at ddpui/tests/integration_tests/test_airbyte_integration.py. This should be a good starting point on this.

We are currently running airbyte 1.4.1 on kuberenetes and doing a local setup for this might require good amout of resources. So It might be difficult to run it locally. Please reach out to me on our discord server and i will give you access to our staging airbyte server.
Docker setup for airbyte after 0.64.x is deprecated and peoplpe have to use either a local or prod kubernetes setup.

Mockups / Wireframes

None

Product Name

Dalgo

Project Name

Dalgo

Organization Name:

Project Tech4Dev

Domain

Data pipeline

Tech Skills Needed:

Integration testing, Pytest, Python, Api integration

Mentor(s)

@Ishankoradia

Complexity

Medium

Category

CI/CD, Integrations, Deployment

Sub Category