Docker build script with chosen connectors

[amigold]

Subject of the issue

I would like to add a bash script to build a docker image that only includes chosen connectors and not default to "all"

The script will ask the user for connectors and output tag and then run the docker build command

May I submit a PR?

[koszti]

would that be a shell script to run the [Dockerfile| https://github.com/transferwise/pipelinewise/blob/master/Dockerfile] and passing a list of connectors to build only the certain components into the image?

Sounds like a nice feature, please go ahead and waiting for the PR :)

[amigold]

Yep, I'm actually imagining a few other improvements down the line:
Following the building (and pushing) of the custom image, it would be great to use Helm to deploy a cronjob on top of Kubernetes. So I'm thinking of building that Helm chart as well

What do you think @koszti?

[koszti]

that sounds amazing really!

[amigold]

@koszti may I also suggest we add a CI script that just takes common tap-target pairs, builds and pushes multiple images

Just so we can have standard images to let people get up to speed faster with the project. We can even iterate on all the supported taps and targets and build the permutations if we really want to...

[amigold]

@koszti ?

[koszti]

hey @amigold, do you mean to start publishing standard, official public docker pipelinewise images automatically to dockerhub for example?

[amigold]

Yes exactly @koszti
At least for the common connector pairings

[koszti]

makes sense, we will discuss it internally in the team and will come back soon with answers. 🙇‍♂️

[amigold]

Ok
I think it will make getting started much faster for most cases

[dirathea]

Hi guys! great idea btw @amigold.
I also using Pipelinewise on my current company and setup it using simple Kustomize YAML templates that run using Jenkins periodically.

And then I think it will be great if we could do it natively, so I created the Kubernetes operator https://github.com/dirathea/pipelinewise-operator (it still WIP btw) to simplify the setup process. And those apps will get a huge advantage if there's an official Docker image for Pipelinewise

[amigold]

Nice one @dirathea ! Great idea
I created a helm chart https://github.com/fairfly/helm-pipelinewise but an operator could definitely be the future

Which sources/targets do you currently support?

[amigold]

Also I think you should use secrets to store the credentials

[dirathea]

I planned to have all the default connection to be supported out of the box.
On the current commit the operator support mysql tap, posrgre target, and redshift target (since MySQL and Postgres are simple to set up and help me test the functionality faster).

The master password as Kubernetes secret also not supported yet, but I think the operator will only access it. The user itself needs to create it manually beforehand.