Skip to content

Commit 0889fba

Browse files
AzureQMarcelo Vanzin
authored and
Marcelo Vanzin
committed
[SPARK-26083][K8S] Add Copy pyspark into corresponding dir cmd in pyspark Dockerfile
When I try to run `./bin/pyspark` cmd in a pod in Kubernetes(image built without change from pyspark Dockerfile), I'm getting an error: ``` $SPARK_HOME/bin/pyspark --deploy-mode client --master k8s://https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT_HTTPS ... Python 2.7.15 (default, Aug 22 2018, 13:24:18) [GCC 6.4.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. Could not open PYTHONSTARTUP IOError: [Errno 2] No such file or directory: '/opt/spark/python/pyspark/shell.py' ``` This is because `pyspark` folder doesn't exist under `/opt/spark/python/` ## What changes were proposed in this pull request? Added `COPY python/pyspark ${SPARK_HOME}/python/pyspark` to pyspark Dockerfile to resolve issue above. ## How was this patch tested? Google Kubernetes Engine Closes apache#23037 from AzureQ/master. Authored-by: Qi Shao <[email protected]> Signed-off-by: Marcelo Vanzin <[email protected]>
1 parent a24e1a1 commit 0889fba

File tree

2 files changed

+2
-0
lines changed

2 files changed

+2
-0
lines changed

bin/docker-image-tool.sh

+1
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,7 @@ function create_dev_build_context {(
107107
"$PYSPARK_CTX/kubernetes/dockerfiles"
108108
mkdir "$PYSPARK_CTX/python"
109109
cp -r "python/lib" "$PYSPARK_CTX/python/lib"
110+
cp -r "python/pyspark" "$PYSPARK_CTX/python/pyspark"
110111

111112
local R_CTX="$CTX_DIR/sparkr"
112113
mkdir -p "$R_CTX/kubernetes"

resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile

+1
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ RUN apk add --no-cache python && \
3838
# Removed the .cache to save space
3939
rm -r /root/.cache
4040

41+
COPY python/pyspark ${SPARK_HOME}/python/pyspark
4142
COPY python/lib ${SPARK_HOME}/python/lib
4243
ENV PYTHONPATH ${SPARK_HOME}/python/lib/pyspark.zip:${SPARK_HOME}/python/lib/py4j-*.zip
4344

0 commit comments

Comments
 (0)