Hadoop Spark:设置conf.warn.jar时出现未知/不支持的参数错误

当我使用Spark submit提交时，我有一个小应用程序在基于纱线的Spark群集上运行良好，如下所示：

~/spark-1.4.0-bin-hadoop2.4$ bin/spark-submit --class MyClass --master yarn-cluster --queue testing myApp.jar hdfs://nameservice1/user/XXX/README.md_count

但是，我不想每次都上传spark-assembly.jar文件，所以我设置了

spark.warn.jar

配置参数：

~/spark-1.4.0-bin-hadoop2.4$ bin/spark-submit --class MyClass --master yarn-cluster --queue testing --conf "spark.yarn.jar=hdfs://nameservice1/user/spark/share/lib/spark-assembly.jar" myApp.jar hdfs://nameservice1/user/XXX/README.md_count

一开始这似乎很好：

15/07/08 13:57:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/07/08 13:57:18 INFO yarn.Client: Requesting a new application from cluster with 24 NodeManagers
15/07/08 13:57:18 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
15/07/08 13:57:18 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
15/07/08 13:57:18 INFO yarn.Client: Setting up container launch context for our AM
15/07/08 13:57:18 INFO yarn.Client: Preparing resources for our AM container
15/07/08 13:57:18 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://nameservice1/user/spark/share/lib/spark-assembly.jar
[...]

然而，它最终失败了：

15/07/08 13:57:18 INFO yarn.Client: Submitting application 670 to ResourceManager
15/07/08 13:57:18 INFO impl.YarnClientImpl: Submitted application application_1434986503384_0670
15/07/08 13:57:19 INFO yarn.Client: Application report for application_1434986503384_0670 (state: ACCEPTED)
15/07/08 13:57:19 INFO yarn.Client: 
 client token: N/A
 diagnostics: N/A
 ApplicationMaster host: N/A
 ApplicationMaster RPC port: -1
 queue: testing
 start time: 1436356638869
 final status: UNDEFINED
 tracking URL: http://node-00a/cluster/app/application_1434986503384_0670
 user: XXX
15/07/08 13:57:20 INFO yarn.Client: Application report for application_1434986503384_0670 (state: ACCEPTED)
15/07/08 13:57:21 INFO yarn.Client: Application report for application_1434986503384_0670 (state: ACCEPTED)
15/07/08 13:57:23 INFO yarn.Client: Application report for application_1434986503384_0670 (state: FAILED)
15/07/08 13:57:23 INFO yarn.Client: 
 client token: N/A
 diagnostics: Application application_1434986503384_0670 failed 2 times due to AM Container for appattempt_1434986503384_0670_000002 exited with  exitCode: 1 due to: Exception from container-launch.
Container id: container_1434986503384_0670_02_000001
Exit code: 1
[...]

在纱线日志中，我发现以下错误消息，表明参数使用错误：

Container: container_1434986503384_0670_01_000001 on node-01b_8041
===================================================================================================
LogType:stderr
Log Upload Time:Mi Jul 08 13:57:22 +0200 2015
LogLength:764
Log Contents:
    Unknown/unsupported param List(--arg, hdfs://nameservice1/user/XXX/README.md_count, --executor-memory, 1024m, --executor-cores, 1, --num-executors, 2)
    Usage: org.apache.spark.deploy.yarn.ApplicationMaster [options] 
Options:
  --jar JAR_PATH       Path to your application's JAR file (required)
  --class CLASS_NAME   Name of your application's main class (required)
  --args ARGS          Arguments to be passed to your application's main class.
                       Mutliple invocations are possible, each will be passed in order.
  --num-executors NUM    Number of executors to start (Default: 2)
  --executor-cores NUM   Number of cores for the executors (Default: 1)
  --executor-memory MEM  Memory per executor (e.g. 1000M, 2G) (Default: 1G)

End of LogType:stderr

由于在提交时上载本地程序集文件时运行相同的应用程序，因此它似乎归结为程序集文件。集群上的版本可能是错误的/不同的版本吗？我怎样才能证实这一点？原因可能是什么？警告

WARN util.NativeCodeLoader:…

是否可能相关

---

当我设置（不推荐使用的）环境变量

SPARK\u JAR

而不是设置

SPARK.warn.JAR

时，也会发生同样的情况。我在这里问了一个明显的问题：您确定HDFS上的SPARK-assembly.JAR与本地的相同吗？如果没有，您可以尝试将您的本地spark程序集上载到HDFS上的主目录，然后重试吗？

应该已经提到过，我也尝试过了，同样的错误。您可以尝试提交spark examples jar并运行一个包含Arument的示例吗？（例如SparkPi）这样我们就可以隔离您的jar是否存在问题。另一个愚蠢的问题是，您是否将所有依赖项打包到了应用程序jar中？如果是这样，spark相关的依赖项被正确地标记为

，前提是

？结果是JVM版本有一些混乱，我无法重现，抱歉。