Apache spark k8s群集中Spark提交失败

我正在尝试使用以下命令将Spark应用程序提交到minikube k8s群集（使用的Spark版本：2.4.3）：

spark-submit \
--master <K8S_MASTER> \
--deploy-mode cluster \
--conf spark.executor.instances=2 \
--conf spark.kubernetes.container.image=<my docker image> \
--conf spark.kubernetes.driver.pod.name=spark-py-driver \
--conf spark.executor.memory=2g \
--conf spark.driver.memory=2g \
local:///home/proj/app/run.py <arguments>

---

我正在使用我组织中的一个基本图像。但关于挂载的问题只是一个警告，吊舱在那之后被成功分配

FROM <project_repo>/<proj>/${SPARK_ALPINE_BUILD}
ENV SPARK_OPTS --driver-java-options=-Dlog4j.logLevel=info
ENV SPARK_MASTER "spark://spark-master:7077"

ADD https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.38/mysql-connector-java-5.1.38.jar $SPARK_HOME/jars
ADD https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.11/2.3.2/spark-cassandra-connector_2.11-2.3.2.jar $SPARK_HOME/jars
USER root

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

WORKDIR /home/<proj>/app
# copy files
COPY src/configs ./configs
COPY src/dependencies ./dependencies
COPY src/jobs ./jobs
COPY src/run.py ./run.py
COPY run.sh ./run.sh 
COPY src/requirements.txt . 

# install packages here 
RUN set -e; \
  pip install --no-cache-dir -r requirements.txt;

FROM/${SPARK\u ALPINE\u BUILD}
ENV SPARK_OPTS——驱动程序java选项=-Dlog4j.logLevel=info
环境星火大师”spark://spark-master:7077"
添加https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.38/mysql-connector-java-5.1.38.jar $SPARK_HOME/jars
添加https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.11/2.3.2/spark-cassandra-connector_2.11-2.3.2.jar $SPARK_HOME/jars
用户根
#设置环境变量
ENV PYTHONDONTWRITEBYTECODE 1
环境蟒蛇缓冲1
WORKDIR/home//app
#复制文件
复制src/configs./configs
复制src/依赖项/依赖项
复制src/jobs./jobs
复制src/run.py./run.py
复制run.sh./run.sh
复制src/requirements.txt。
#在此处安装软件包
运行集-e\
pip安装——无缓存目录-r requirements.txt；

使用的spark submit不是3.0.0版本。您还需要将使用spark submit的spark安装更改为3.0.0版。

能否共享docker文件？看起来您没有在应用程序的容器中装入卷。仅供将来参考-如果您提供了更多信息，请不要再给出其他答案。编辑原始帖子以添加更多信息。

FROM <project_repo>/<proj>/${SPARK_ALPINE_BUILD}
ENV SPARK_OPTS --driver-java-options=-Dlog4j.logLevel=info
ENV SPARK_MASTER "spark://spark-master:7077"

ADD https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.38/mysql-connector-java-5.1.38.jar $SPARK_HOME/jars
ADD https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.11/2.3.2/spark-cassandra-connector_2.11-2.3.2.jar $SPARK_HOME/jars
USER root

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

WORKDIR /home/<proj>/app
# copy files
COPY src/configs ./configs
COPY src/dependencies ./dependencies
COPY src/jobs ./jobs
COPY src/run.py ./run.py
COPY run.sh ./run.sh 
COPY src/requirements.txt . 

# install packages here 
RUN set -e; \
  pip install --no-cache-dir -r requirements.txt;