Skip to content

Commit

Permalink
kubernetes local should work with in process launcher (#601)
Browse files Browse the repository at this point in the history
## Upstream SPARK-XXXXX ticket and PR link (if not applicable, explain)

local kubernetes submission is not submitted upstream yet

## What changes were proposed in this pull request?

In process launcher clobbers spark configuration with launcher port and secret overrides, resulting the local spark submit on the driver to connect to the port used by the cluster mode spark submit process.

Spark drivers in k8s are hard coded to run spark-submit, so spark launcher overrides are inapplicable to them. Excluding the configurations from the config map properly signals the local backend to not try to connect to a launcher server
  • Loading branch information
onursatici authored and bulldozer-bot[bot] committed Aug 29, 2019
1 parent a7df826 commit 3d3dacb
Showing 1 changed file with 3 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,9 @@ private[spark] class Client(
// Build a Config Map that will house spark conf properties in a single file for spark-submit
private def buildConfigMap(configMapName: String, conf: Map[String, String]): ConfigMap = {
val properties = new Properties()
conf.foreach { case (k, v) =>
conf
.filter { case (k, _) => !k.startsWith("spark.launcher")}
.foreach { case (k, v) =>
properties.setProperty(k, v)
}
val propertiesWriter = new StringWriter()
Expand Down

0 comments on commit 3d3dacb

Please sign in to comment.