[SPARK-43504][K8S] Mounts the hadoop config map on the executor pod #41181

turboFei · 2023-05-16T03:52:31Z

What changes were proposed in this pull request?

In this pr, for spark on k8s, the hadoop config map will be mounted in executor side as well.
Before, the hadoop config map is only mounted in driver side.

Why are the changes needed?

Since SPARK-25815 , the hadoop config map will not be mounted in executor side.

Per the #22911 description:

The main two things that don't need to happen in executors anymore are:

adding the Hadoop config to the executor pods: this is not needed
since the Spark driver will serialize the Hadoop config and send
it to executors when running tasks.

But in fact, the executor still need the hadoop configuration.

As shown in above picture, the driver can resolve hdfs://zeus, but the executor can not.

so we still need to mount the hadoop config map in executor side.

Does this PR introduce any user-facing change?

Yes, users do not need to take workarounds to make executors load the hadoop configuration.
Such as:

including hadoop conf in executor image
placing hadoop conf files under SPARK_CONF_DIR.

How was this patch tested?

UT.

pan3793 · 2023-05-16T05:38:20Z

I encountered the same issues w/ Spark 3.3.1. This sounds like a regression (I suppose it works before SPARK-25815, I don't have experience w/ running such an old version of Spark on K8s).

The key point is that the executor needs to download artifacts during the bootstrap phase, so the assumption in SPARK-25815 is not always true.

adding the Hadoop config to the executor pods: this is not needed
since the Spark driver will serialize the Hadoop config and send
it to executors when running tasks.

Given the executor use SparkHadoopUtil.get.newConfiguration(conf) to construct Hadoop conf, we can put the related hdfs/s3 configurations into spark-defaults.conf w/ spark.hadoop. prefix as a workaround.

spark/core/src/main/scala/org/apache/spark/executor/Executor.scala

Lines 1006 to 1012 in 0df4c01

    
           private[executor] def updateDependencies( 
        
               newFiles: Map[String, Long], 
        
               newJars: Map[String, Long], 
        
               newArchives: Map[String, Long], 
        
               testStartLatch: Option[CountDownLatch] = None, 
        
               testEndLatch: Option[CountDownLatch] = None): Unit = { 
        
             lazy val hadoopConf = SparkHadoopUtil.get.newConfiguration(conf)

This PR definitely fixes some use cases, @turboFei would you mind updating "Does this PR introduce any user-facing change?"

turboFei · 2023-05-16T05:45:27Z

would you mind updating "Does this PR introduce any user-facing change?"

updated

...core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala

turboFei · 2023-05-16T14:46:23Z

seems the k8s integration testing is stuck, will check this pr in our dev hadoop cluster tomorrow.

dongjoon-hyun

Thank you for making a PR, @turboFei .

However, this PR might cause a outage because the number of configMap is controlled by quota.

$ kubectl describe quota | grep configmaps
count/configmaps                                                  4     150

To avoid the production outage, this should be under a new configuration with false by default at least.

turboFei · 2023-05-17T06:07:00Z

thanks for the comments, I will check it

advancedxy · 2023-05-17T06:13:24Z

Thank you for making a PR, @turboFei .

However, this PR might cause a outage because the number of configMap is controlled by quota.
$ kubectl describe quota | grep configmaps
count/configmaps                                                  4     150
To avoid the production outage, this should be under a new configuration with false by default at least.

150 is a bit small for serious production usage, we may add this note in the running_on_k8s.md documentation.

And BTW, this PR doesn't create new ConfigMaps, it either uses a user pre-set config map (no creation) or just reuse the config map created by driver which is created if necessary.

turboFei · 2023-05-17T08:29:53Z

this PR doesn't create new ConfigMaps, it either uses a user pre-set config map (no creation) or just reuse the config map created by driver which is created if necessary.

yes, this PR doesn't create new ConfigMap.

dongjoon-hyun · 2023-05-17T08:35:00Z

Oh, got it. Thank you for correcting me.

turboFei · 2023-05-17T10:10:40Z

the UT has passed, gentle ping @dongjoon-hyun

...src/test/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStepSuite.scala

advancedxy

lgtm.
left one minor comment, I'm ok with merging this as it is.

Stale

turboFei · 2023-05-18T03:53:59Z

gentle ping @dongjoon-hyun would you like to review again? thanks

yaooqinn · 2023-05-19T06:14:03Z

The Hadoop configurations can be propagated after #27735. And putting and locating extra configuration files in SPARK_HOME/conf is also a suggested way from our docs, so is this step necessary?

Alternatively, if both exist, what is the precedence between them? Is it idempotent?

turboFei · 2023-05-22T03:05:18Z

And putting and locating extra configuration files in SPARK_HOME/conf is also a suggested way from our docs, so is this step necessary?

I think it is necessary.

Hadoop and spark are different components, it is better to maintain them separately.

In our company, we have conf version for hadoop conf, so we do not put hadoop config files under SPARK_HOME/conf, we use soft link to manage the hadoop conf.

Alternatively, if both exist, what is the precedence between them? Is it idempotent?

In this pr, it just mounts the hadoop config map in the executor side(mounts HADOOP_CONF_DIR env) and the hadoop conf mounted is absolute same with that in driver pod.

As shown below, the SPARK_CONF_DIR has higher precedence. I think it is idempotent.

spark/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh

Lines 68 to 76 in e096bce

    
           if ! [ -z ${HADOOP_CONF_DIR+x} ]; then 
        
             SPARK_CLASSPATH="$HADOOP_CONF_DIR:$SPARK_CLASSPATH"; 
        
           fi 
        
           if ! [ -z ${SPARK_CONF_DIR+x} ]; then 
        
             SPARK_CLASSPATH="$SPARK_CONF_DIR:$SPARK_CLASSPATH"; 
        
           elif ! [ -z ${SPARK_HOME+x} ]; then 
        
             SPARK_CLASSPATH="$SPARK_HOME/conf:$SPARK_CLASSPATH"; 
        
           fi

turboFei · 2023-05-23T03:55:37Z

gentle ping @yaooqinn @dongjoon-hyun

yaooqinn · 2023-05-23T04:09:12Z

Hadoop and spark are different components, it is better to maintain them separately.

I do not fully agree. In the early days, Hadoop may be special. We have a specific code path to read HADOOP_CONF_DIR. But now Hadoop is an option, as we have other options for storage and scheduling, especially on the cloud or Kubernetes.

Maybe we shall treat it like hive configurations or other third-party components to reduce the maintenance burden and complexity of the deployment.

turboFei · 2023-05-23T04:35:07Z

Maybe we shall treat it like hive configurations or other third-party components to reduce the maintenance burden and complexity of the deployment.

I believe different companies treat hadoop conf differently.

For ebay, we add conf version for hadoop conf, because it is used by

public hadoop client nodes
private hadoop client nodes
hadoop service nodes(nn, rm, hms, kyuubi)
hadoop slave nodes(nm, dn)

and between different conf versions, there might be incompatibilities.

and we have an RESTful service to download the hdaoop conf and we use soft link to manage them locally.

Recently, we are making spark migration, from spark3 + hadoop2 to spark3 + hadoop3.

For hadoop2 and hadoop3, the hadoop confs are even different.

So to manage the hadoop conf well and due to the current situation, in ebay, we do not want to put the hadoop conf files and spark conf files together.

treat it like hive configurations or other third-party components to reduce the maintenance burden and complexity of the deployment.

yes, I agree, it makes it easy.

infoankitp · 2023-05-25T04:40:52Z

I can also help with a use case for this, usually the submission client is on a single environment (Lets say we have it on cloud), and with spark on k8s, we can easily run jobs in different envs like in private Cloud Clusters being submitted from public Cloud. Where we would need diff properties to be passed for the submission client as well as for drivers and executors. This is also a use case where mounting the hadoopConfMap in executors would help in making the task easy to maintain the configs.

turboFei · 2023-05-25T07:59:07Z

I can also help with a use case for this, usually the submission client is on a single environment (Lets say we have it on cloud), and with spark on k8s, we can easily run jobs in different envs like in private Cloud Clusters being submitted from public Cloud. Where we would need diff properties to be passed for the submission client as well as for drivers and executors. This is also a use case where mounting the hadoopConfMap in executors would help in making the task easy to maintain the configs.

Yes, I think this pr is general for hadoop conf use case, and it does not create more resource because it just use the existing config map.

@yaooqinn @dongjoon-hyun could you help to take another look? Appreciated for your help.

dongjoon-hyun

Although this way is a little controversial like @yaooqinn pointed out, I agree with the intention and use-cases and I believe we can allow this way back additionally without many burden or intervention because spark.kubernetes.executor.hadoopConfigMapName is still reserved in the code.

If there is no strong objection from other people, +1 from my side. Please let us know if you still disagree, @yaooqinn .

dongjoon-hyun · 2023-06-01T04:30:32Z

Merged to master for Apache Spark 3.5.0.

turboFei · 2023-06-01T04:40:06Z

thanks all !!!

yaooqinn · 2023-06-05T03:45:11Z

thanks, @dongjoon-hyun and @turboFei. Late +1 from my side.

### What changes were proposed in this pull request? In this pr, for spark on k8s, the hadoop config map will be mounted in executor side as well. Before, the hadoop config map is only mounted in driver side. ### Why are the changes needed? Since [SPARK-25815](https://issues.apache.org/jira/browse/SPARK-25815) [,](apache#22911,) the hadoop config map will not be mounted in executor side. Per the apache#22911 description: > The main two things that don't need to happen in executors anymore are: > 1. adding the Hadoop config to the executor pods: this is not needed > since the Spark driver will serialize the Hadoop config and send > it to executors when running tasks. But in fact, the executor still need the hadoop configuration. ![image](https://github.com/apache/spark/assets/6757692/ff6374c9-7ebd-4472-a85c-99c75a737e2a) As shown in above picture, the driver can resolve `hdfs://zeus`, but the executor can not. so we still need to mount the hadoop config map in executor side. ### Does this PR introduce _any_ user-facing change? Yes, users do not need to take workarounds to make executors load the hadoop configuration. Such as: - including hadoop conf in executor image - placing hadoop conf files under `SPARK_CONF_DIR`. ### How was this patch tested? UT. Closes apache#41181 from turboFei/exec_hadoop_conf. Authored-by: fwang12 <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

maomaodev · 2025-01-07T08:42:07Z

gentle ping @turboFei
In the cluster mode, this merge has caused a new issue in our environment.
After the executor mounts the Hadoop conf file and tries to retrieve user information during startup, it fails to start because the conf file contains Kerberos authentication configuration, but the executor does not mount krb5.conf. Have you tested this scenario?

pan3793 · 2025-01-07T09:08:37Z

@maomaodev Generally, the executor is supposed to use the delegation token instead of keytab to access kerberized HDFS. I wonder how do you set up your Kerberos auth for Spark? And what's the behavior before this PR.

turboFei · 2025-01-07T09:12:48Z

gentle ping @turboFei In the cluster mode, this merge has caused a new issue in our environment. After the executor mounts the Hadoop conf file and tries to retrieve user information during startup, it fails to start because the conf file contains Kerberos authentication configuration, but the executor does not mount krb5.conf. Have you tested this scenario?

How about uploading the krb5.conf?

spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

Lines 572 to 579 in 36d23ef

    
           val KUBERNETES_KERBEROS_KRB5_FILE = 
        
             ConfigBuilder("spark.kubernetes.kerberos.krb5.path") 
        
               .doc("Specify the local location of the krb5.conf file to be mounted on the driver " + 
        
                 "and executors for Kerberos. Note: The KDC defined needs to be " + 
        
                 "visible from inside the containers ") 
        
               .version("3.0.0") 
        
               .stringConf 
        
               .createOptional

maomaodev · 2025-01-07T09:35:58Z

@maomaodev Generally, the executor is supposed to use the delegation token instead of keytab to access kerberized HDFS. I wonder how do you set up your Kerberos auth for Spark? And what's the behavior before this PR.

Yes, the executor does use delegation token, but based on the stacktrace provided above, the code is throwing an exception before adding the delegation token.

We use Kyuubi for submission, using the parameter spark.kubernetes.kerberos.krb5.path.

Prior to this PR, we were using Spark 3.4.2, and the executor did not mount the Hadoop conf during startup. The executor logs showed simple authentication.

maomaodev · 2025-01-07T09:39:33Z

spark.kubernetes.kerberos.krb5.path

The parameter(spark.kubernetes.kerberos.krb5.path) has been set.

pan3793 · 2025-01-07T10:32:44Z

@maomaodev I see what happens, the official Spark image installs krb5-user which generates a default /etc/krb5.conf with content

$ docker run --rm apache/spark:3.5.4 cat /etc/krb5.conf
[libdefaults]
	default_realm = ATHENA.MIT.EDU
...

so that KerberosUtil.getDefaultRealm works well, this is not the expected behavior but happens to work. I think the root cause is spark.kubernetes.kerberos.krb5.path claims to mount on both driver and executors but actually not.

In short, creating a dummy /etc/krb5.conf in your base image should workaround your issue, and the correct solution is mounting spark.kubernetes.kerberos.krb5.path to executor pod correctly.

maomaodev · 2025-01-07T11:15:13Z

@maomaodev I see what happens, the official Spark image installs krb5-user which generates a default /etc/krb5.conf with content
$ docker run --rm apache/spark:3.5.4 cat /etc/krb5.conf
[libdefaults]
	default_realm = ATHENA.MIT.EDU
...
so that KerberosUtil.getDefaultRealm works well, this is not the expected behavior but happens to work. I think the root cause is spark.kubernetes.kerberos.krb5.path claims to mount on both driver and executors but actually not.

In short, creating a dummy /etc/krb5.conf in your base image should workaround your issue, and the correct solution is mounting spark.kubernetes.kerberos.krb5.path to executor pod correctly.

Yes, creating a dummy /etc/krb5.conf in the base image does work. Is the community planning to fix this issue?

pan3793 · 2025-01-07T11:48:10Z

@maomaodev I will prepare a PR soon

maomaodev · 2025-01-07T14:41:43Z

@maomaodev I will prepare a PR soon

Thank you, I have created a corresponding jira：https://issues.apache.org/jira/browse/SPARK-50758

turboFei added 2 commits May 16, 2023 11:08

mount hadoop conf volume for executor

c9c3ce5

add feature to builder

f881767

turboFei changed the title ~~[SPARK-43504] Mount hadoop config map in executor side~~ [SPARK-43504][K8S] Mount hadoop config map in executor side May 16, 2023

github-actions bot added the KUBERNETES label May 16, 2023

turboFei added 2 commits May 16, 2023 12:02

add ut

7190f93

refactor

c599d24

turboFei force-pushed the exec_hadoop_conf branch from 2945219 to c599d24 Compare May 16, 2023 04:08

turboFei changed the title ~~[SPARK-43504][K8S] Mount hadoop config map in executor side~~ [SPARK-43504][K8S] Mount hadoop config map on the executor pod May 16, 2023

refactor

6689b74

turboFei changed the title ~~[SPARK-43504][K8S] Mount hadoop config map on the executor pod~~ [SPARK-43504][K8S] Mounts the hadoop config map on the executor pod May 16, 2023

turboFei force-pushed the exec_hadoop_conf branch from 2b052b1 to 6689b74 Compare May 16, 2023 04:22

check executor disable config map

1b5799b

advancedxy reviewed May 16, 2023

View reviewed changes

...core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala Outdated Show resolved Hide resolved

advancedxy reviewed May 16, 2023

View reviewed changes

...core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala Outdated Show resolved Hide resolved

turboFei added 2 commits May 16, 2023 20:31

comment

7838785

code style

6fc791f

dongjoon-hyun previously requested changes May 17, 2023

View reviewed changes

fix ut

008846b

turboFei requested review from advancedxy and dongjoon-hyun May 17, 2023 10:11

advancedxy reviewed May 17, 2023

View reviewed changes

...src/test/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStepSuite.scala Outdated Show resolved Hide resolved

advancedxy approved these changes May 17, 2023

View reviewed changes

advancedxy approved these changes May 18, 2023

View reviewed changes

dongjoon-hyun approved these changes May 30, 2023

View reviewed changes

dongjoon-hyun closed this in 10ee643 Jun 1, 2023

turboFei deleted the exec_hadoop_conf branch June 1, 2023 04:39

This was referenced Jan 9, 2025

[SPARK-50758][K8S]Mounts the krb5 config map on the executor pod #49426

Closed

[SPARK-50758][K8S]Mounts the krb5 config map on the executor pod #49467

Closed

[SPARK-43504][K8S] Mounts the hadoop config map on the executor pod #41181

[SPARK-43504][K8S] Mounts the hadoop config map on the executor pod #41181

Uh oh!

Conversation

turboFei commented May 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

pan3793 commented May 16, 2023

Uh oh!

turboFei commented May 16, 2023

Uh oh!

Uh oh!

Uh oh!

turboFei commented May 16, 2023

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

turboFei commented May 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

advancedxy commented May 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

turboFei commented May 17, 2023

Uh oh!

dongjoon-hyun commented May 17, 2023

Uh oh!

turboFei commented May 17, 2023

Uh oh!

Uh oh!

advancedxy left a comment

Choose a reason for hiding this comment

Uh oh!

turboFei commented May 18, 2023

Uh oh!

yaooqinn commented May 19, 2023

Uh oh!

turboFei commented May 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

turboFei commented May 23, 2023

Uh oh!

yaooqinn commented May 23, 2023

Uh oh!

turboFei commented May 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

infoankitp commented May 25, 2023

Uh oh!

turboFei commented May 25, 2023

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jun 1, 2023

Uh oh!

turboFei commented Jun 1, 2023

Uh oh!

yaooqinn commented Jun 5, 2023

Uh oh!

maomaodev commented Jan 7, 2025

Uh oh!

pan3793 commented Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

turboFei commented Jan 7, 2025

Uh oh!

maomaodev commented Jan 7, 2025

Uh oh!

maomaodev commented Jan 7, 2025

Uh oh!

pan3793 commented Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

turboFei commented May 16, 2023 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

turboFei commented May 17, 2023 •

edited

Loading

advancedxy commented May 17, 2023 •

edited

Loading

turboFei commented May 22, 2023 •

edited

Loading

turboFei commented May 23, 2023 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

pan3793 commented Jan 7, 2025 •

edited

Loading

pan3793 commented Jan 7, 2025 •

edited

Loading