Apache spark 纱线簇上的火花流

我有一个由1个主节点和2个从节点组成的集群。我正在master中运行spark流，我想利用集群中的所有节点。我在代码中指定了一些参数，如驱动程序内存和执行器内存。当我在spark submit中给出--deploy mode cluster--master warn cluster时，它给出了以下错误

> log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/08/12 13:24:49 INFO Client: Requesting a new application from cluster with 3 NodeManagers
15/08/12 13:24:49 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
15/08/12 13:24:49 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
15/08/12 13:24:49 INFO Client: Setting up container launch context for our AM
15/08/12 13:24:49 INFO Client: Preparing resources for our AM container
15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/assembly/target/scala-2.10/spark-assembly-1.4.1-hadoop2.5.0-cdh5.3.5.jar
15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/external/kafka-assembly/target/spark-streaming-kafka-assembly_2.10-1.4.1.jar
15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip
15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/python/lib/py4j-0.8.2.1-src.zip
15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/examples/src/main/python/streaming/kyt.py
15/08/12 13:24:49 INFO Client: Setting up the launch environment for our AM container
15/08/12 13:24:49 INFO SecurityManager: Changing view acls to: hdfs
15/08/12 13:24:49 INFO SecurityManager: Changing modify acls to: hdfs
15/08/12 13:24:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hdfs); users with modify permissions: Set(hdfs)
15/08/12 13:24:49 INFO Client: Submitting application 3808 to ResourceManager
15/08/12 13:24:49 INFO YarnClientImpl: Submitted application application_1437639737006_3808
15/08/12 13:24:50 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:24:50 INFO Client: 
   client token: N/A
   diagnostics: N/A
   ApplicationMaster host: N/A
   ApplicationMaster RPC port: -1
   queue: root.hdfs
   start time: 1439385889600
   final status: UNDEFINED
   tracking URL: http://hostname:port/proxy/application_1437639737006_3808/
   user: hdfs
15/08/12 13:24:51 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:24:52 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:24:53 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:24:54 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:24:55 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:24:56 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:24:57 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:24:58 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:24:59 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:25:00 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:25:01 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:25:02 INFO Client: Application report for application_1437639737006_3808 (state: ACCEPTED)
15/08/12 13:25:03 INFO Client: Application report for application_1437639737006_3808 (state: FAILED)
15/08/12 13:25:03 INFO Client: 
   client token: N/A
   diagnostics: Application application_1437639737006_3808 failed 2 times due to AM Container for appattempt_1437639737006_3808_000002 exited with  exitCode: -1000 due to: File file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip does not exist
.Failing this attempt.. Failing the application.
   ApplicationMaster host: N/A
   ApplicationMaster RPC port: -1
   queue: root.hdfs
   start time: 1439385889600
   final status: FAILED
   tracking URL: http://hostname:port/cluster/app/application_1437639737006_3808
   user: hdfs
Exception in thread "main" org.apache.spark.SparkException: Application application_1437639737006_3808 finished with failed status
  at org.apache.spark.deploy.yarn.Client.run(Client.scala:855)
  at org.apache.spark.deploy.yarn.Client$.main(Client.scala:881)
  at org.apache.spark.deploy.yarn.Client.main(Client.scala)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
  at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
  at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

---

如何解决这个问题？如果我做错了，请帮助我。

在纱线簇模式下运行时，您始终需要为执行器指定其他内存设置，并且需要单独指定内存，另外还需要指定驱动程序详细信息。比如现在

Amazon EC2环境（已保留）：

m3.xlarge |核心：4（1）| RAM：15（3.5）|硬盘：80 GB |节点：3个节点
spark submit--class--master纱线簇--num executors 2--driver memory 8g--executor memory 8g--executor cores 1

始终记住将其他第三方库或JAR添加到每个任务节点的类路径中，您可以直接将它们添加到每个节点上的Spark或Hadoop类路径中

注: 1） 如果您使用的是AmazonEMR，那么它可以通过自定义引导操作和S3实现。 2） 也移除冲突的罐子。有时您会看到一个不必要的NullPointerException，这可能是其关键原因之一

如果可能，请使用添加stacktrace

yarn logs -applicationId <HadoopAppId>

纱线原木-应用程序ID

---

所以我可以用更具体的方式回答你。

我最近遇到了同样的问题。以下是我的设想：

Cloudera管理的CDH 5.3.3群集有7个节点。我是从其中一个节点提交作业的，它在“纱线簇”和“纱线主控”模式下都会出现相同的问题

如果你看一下stacktrace，你会发现这一行-

15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/external/kafka-assembly/target/spark-streaming-kafka-assembly_2.10-1.4.1.jar
15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip
15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/python/lib/py4j-0.8.2.1-src.zip
15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/examples/src/main/python/streaming/kyt.py

这就是作业失败的原因，因为没有复制资源

---

在我的例子中，通过更正HADOOP_CONF_DIR路径解决了这个问题。它没有指向包含core-site.xml和warn-site.xml以及其他配置文件的确切文件夹。修复后，在启动ApplicationMaster的过程中复制了资源，并且作业正常运行。

您提交的文件：

/home/hdfs/spark-1.4.1/python/lib/pyspark.zip

不存在。

我可以通过在运行时提供驱动程序内存和执行程序内存来解决这个问题

---

spark submit--驱动程序内存1g--执行器内存1g--class com.package.App--master Thread--部署模式cluster/home/spark.jar

能否添加运行程序的确切命令？能否从Thread日志中添加StackTrace-applicationId。以便我能更具体地回答

15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/external/kafka-assembly/target/spark-streaming-kafka-assembly_2.10-1.4.1.jar
15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip
15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/python/lib/py4j-0.8.2.1-src.zip
15/08/12 13:24:49 INFO Client: Source and destination file systems are the same. Not copying file:/home/hdfs/spark-1.4.1/examples/src/main/python/streaming/kyt.py