You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, i'm currently working with DBT on a ClickHouse cluster and I wondered if it is possible, with a single invocation of the dbt cli, to launch it with multiple profiles. I would want to "distribute" every materializations on my cluster to increase performances and use the maximum of the dedicated hardware. In my cluster, everything is replicated, let's say we have 4 marts A, B, C, D: I would want to have A and B materialized on node ch-01 and C and D on node ch-02, the replication is assured by ClickHouse.
To try to achieve this I setup a round robin DNS behind my domain name, that covers 2 nodes of my cluster, let's say ch-01 and ch-02. At the invocation of the CLI, DBT select the given profile and use the associated hostname, but, for the whole invocation, it is run on ch-01 OR ch-02, and does not use the two ones... It seems that DBT open a unique connection and keep it through the pipeline.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello, i'm currently working with DBT on a ClickHouse cluster and I wondered if it is possible, with a single invocation of the dbt cli, to launch it with multiple profiles. I would want to "distribute" every materializations on my cluster to increase performances and use the maximum of the dedicated hardware. In my cluster, everything is replicated, let's say we have 4 marts A, B, C, D: I would want to have A and B materialized on node ch-01 and C and D on node ch-02, the replication is assured by ClickHouse.
To try to achieve this I setup a round robin DNS behind my domain name, that covers 2 nodes of my cluster, let's say ch-01 and ch-02. At the invocation of the CLI, DBT select the given profile and use the associated hostname, but, for the whole invocation, it is run on ch-01 OR ch-02, and does not use the two ones... It seems that DBT open a unique connection and keep it through the pipeline.
Is there a way to achieve my thought ?
Thanks you !
Beta Was this translation helpful? Give feedback.
All reactions