Environment variable problem

[novice-gamer]

The dolphin scheduler is about the configuration of big data environment variables. How should I configure them if they are distributed on different machines? For example, the big data cluster is on the cloud, and the dolphin scheduler is deployed on the local machine.

[zhongjiajie]

could you give more detail about what task type you want to use, for example, if you use hive, you could create hive data source connection, and the enter the jdbc parameter you remote machine is and run it

[novice-gamer]

sqoop, spark and hive are all used.

[novice-gamer]

Whether to configure it in the data source center is OK. There is no need to configure environment variables in the "Dolphin scheduler_env.sh" file

[zhongjiajie]

dolphinscheduler connect hive via jdbc, and they could use remote connect. But both sqoop and spark task should run with dolphinscheduler's worker, so you have to install worker to you cluster. you could create a new worker group to run specific task like sqoop or spark

[novice-gamer]

ok, thank you

[zhongjiajie]

you welcome

[yxsgao]

lets say if I deployed dolphin scheduler on cluster A , in cluster A, there is no Hadoop or hive.
and I wish to use dolphin scheduler on cluster A to run cluster B . Cluster B has Hadoop, spark, hive and etc.

1. How should I configure the environment to use ds cluster A to run big cluster B?
   If I get this, in case of spark, I should also deployed workers on the cluster B?
   how about hive ?
2. Do I have to deployed Hadoop /hive/spark on cluster A ?

Really appreciated if you guys could help, looking forward for you replies.

[zhongjiajie]

You should do that with two ways

• deploy dolphinscheduler worker in your big data cluster B
• make your service can be call remote and call them from cluster A. like hive server2 or make spark-submit work in cluster A, HDFS could have remote connect too

[yxsgao]

Thanks :>