Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/6.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Apache spark 火花:容器已启动,但未开始作业_Apache Spark - Fatal编程技术网

Apache spark 火花:容器已启动,但未开始作业

Apache spark 火花:容器已启动,但未开始作业,apache-spark,Apache Spark,我有一份星火般的工作,起步不错,但所有executor container都显示: INFO YarnAllocator: Received 1 containers from YARN, launching executors on 0 of them. mapreduce作业如下所示: >>> Invoking Shell command line now >> Stdoutput spark.driver.cores is set but does not

我有一份星火般的工作,起步不错,但所有executor container都显示:

INFO YarnAllocator: Received 1 containers from YARN, launching executors on 0 of them.
mapreduce作业如下所示:

>>> Invoking Shell command line now >>

Stdoutput spark.driver.cores is set but does not apply in client mode.
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
而且它没有开始工作。我试着将spark.executor.instances减少到1,它可以工作,但当我改回4时,它会像这样挂起

我认为这是一个配置问题,我应该怎么做? 我的配置:

spark.driver.cores=1
spark.driver.memory=2g
spark.yarn.am.cores=1
spark.yarn.am.memory=4096m
spark.executor.instances=4
spark.executor.cores=4
spark.executor.memory=2g

我认为您应该检查spark cluster是否可以正常提交简单作业,例如:

./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
--driver-memory 1g \
--executor-memory 1g \
--executor-cores 1 \
examples/jars/spark-examples*.jar \
10