[SPARK-22466][SPARK SUBMIT] export SPARK_CONF_DIR while conf is default

## What changes were proposed in this pull request?

We use SPARK_CONF_DIR to switch spark conf directory and can be visited  if we explicitly export it in spark-env.sh, but with default settings, it can't be done. This PR export SPARK_CONF_DIR while it is default.

### Before

```
KentKentsMacBookPro  ~/Documents/spark-packages/spark-2.3.0-SNAPSHOT-bin-master  bin/spark-shell --master local
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/11/08 10:28:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/08 10:28:45 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Spark context Web UI available at http://169.254.168.63:4041
Spark context available as 'sc' (master = local, app id = local-1510108125770).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.3.0-SNAPSHOT
      /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_65)
Type in expressions to have them evaluated.
Type :help for more information.

scala> sys.env.get("SPARK_CONF_DIR")
res0: Option[String] = None
```

### After

```
scala> sys.env.get("SPARK_CONF_DIR")
res0: Option[String] = Some(/Users/Kent/Documents/spark/conf)
```
## How was this patch tested?

vanzin

Author: Kent Yao <[email protected]>

Closes apache#19688 from yaooqinn/SPARK-22466.

Author: yaooqinn

bin/load-spark-env.cmd

@@ -19,15 +19,13 @@ rem
 
 rem This script loads spark-env.cmd if it exists, and ensures it is only loaded once.
 rem spark-env.cmd is loaded from SPARK_CONF_DIR if set, or within the current directory's
-rem conf/ subdirectory.
+rem conf\ subdirectory.
 
 if [%SPARK_ENV_LOADED%] == [] (
   set SPARK_ENV_LOADED=1
 
-  if not [%SPARK_CONF_DIR%] == [] (
-    set user_conf_dir=%SPARK_CONF_DIR%
-  ) else (
-    set user_conf_dir=..\conf
+  if [%SPARK_CONF_DIR%] == [] (
+    set SPARK_CONF_DIR=%~dp0..\conf
   )
 
   call :LoadSparkEnv
@@ -54,6 +52,6 @@ if [%SPARK_SCALA_VERSION%] == [] (
 exit /b 0
 
 :LoadSparkEnv
-if exist "%user_conf_dir%\spark-env.cmd" (
-  call "%user_conf_dir%\spark-env.cmd"
+if exist "%SPARK_CONF_DIR%\spark-env.cmd" (
+  call "%SPARK_CONF_DIR%\spark-env.cmd"
 )

bin/load-spark-env.sh

@@ -29,15 +29,12 @@ fi
 if [ -z "$SPARK_ENV_LOADED" ]; then
   export SPARK_ENV_LOADED=1
 
-  # Returns the parent of the directory this script lives in.
-  parent_dir="${SPARK_HOME}"
+  export SPARK_CONF_DIR="${SPARK_CONF_DIR:-"${SPARK_HOME}"/conf}"
 
-  user_conf_dir="${SPARK_CONF_DIR:-"$parent_dir"/conf}"
-
-  if [ -f "${user_conf_dir}/spark-env.sh" ]; then
+  if [ -f "${SPARK_CONF_DIR}/spark-env.sh" ]; then
     # Promote all variable declarations to environment (exported) variables
     set -a
-    . "${user_conf_dir}/spark-env.sh"
+    . "${SPARK_CONF_DIR}/spark-env.sh"
     set +a
   fi
 fi

conf/spark-env.sh.template

@@ -32,7 +32,8 @@
 # - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data
 # - MESOS_NATIVE_JAVA_LIBRARY, to point to your libmesos.so if you use Mesos
 
-# Options read in YARN client mode
+# Options read in YARN client/cluster mode
+# - SPARK_CONF_DIR, Alternate conf dir. (Default: ${SPARK_HOME}/conf)
 # - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
 # - YARN_CONF_DIR, to point Spark towards YARN configuration files when you use YARN
 # - SPARK_EXECUTOR_CORES, Number of cores for the executors (Default: 1).