在Cloudera或EMR上部署Spark Scala
我创建了一个示例应用程序（代码片段1），我使用（sbt assembly）将其部署到Cloudera上，它可以正常工作在Cloudera或EMR上部署Spark Scala,scala,apache-spark,deployment,cloudera,emr,Scala,Apache Spark,Deployment,Cloudera,Emr,我创建了一个示例应用程序（代码片段1），我使用（sbt assembly）将其部署到Cloudera上，它可以正常工作 object Main { def main(args: Array[String]) { val conf = new SparkConf().setAppName("Spark Pi") val spark = new SparkContext(conf) val slices = if (args.length > 0) args(0
object Main {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Spark Pi")
val spark = new SparkContext(conf)
val slices = if (args.length > 0) args(0).toInt else 2
val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
val count = spark
.parallelize(1 until n, slices)
.map { i =>
val x = random * 2 - 1
val y = random * 2 - 1
if (x * x + y * y < 1) 1 else 0
}
.reduce(_ + _)
println("Pi is roughly " + 4.0 * count / n)
spark.stop()
}
}
spark-submit --master yarn-cluster path_to_jar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/flume-ng/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/07/20 04:15:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/07/20 04:15:36 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/07/20 04:15:36 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers
17/07/20 04:15:36 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/07/20 04:15:36 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
17/07/20 04:15:36 INFO yarn.Client: Setting up container launch context for our AM
17/07/20 04:15:36 INFO yarn.Client: Setting up the launch environment for our AM container
17/07/20 04:15:36 INFO yarn.Client: Preparing resources for our AM container
17/07/20 04:15:38 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
17/07/20 04:15:38 INFO yarn.Client: Uploading resource file:/usr/lib/spark/lib/spark-assembly-1.6.0-cdh5.10.0-hadoop2.6.0-cdh5.10.0.jar -> hdfs://quickstart.cloudera:8020/user/cloudera/.sparkStaging/application_1500545966176_0008/spark-assembly-1.6.0-cdh5.10.0-hadoop2.6.0-cdh5.10.0.jar
17/07/20 04:15:39 INFO yarn.Client: Uploading resource file:/media/sf_SparkS3AirFlow/target/scala-2.11/spark-emr-test.jar -> hdfs://quickstart.cloudera:8020/user/cloudera/.sparkStaging/application_1500545966176_0008/spark-emr-test.jar
17/07/20 04:15:41 INFO yarn.Client: Uploading resource file:/tmp/spark-3a2fec6b-6148-49a9-9db7-7a7cca99c586/__spark_conf__2098882718740574102.zip -> hdfs://quickstart.cloudera:8020/user/cloudera/.sparkStaging/application_1500545966176_0008/__spark_conf__2098882718740574102.zip
17/07/20 04:15:41 INFO spark.SecurityManager: Changing view acls to: cloudera
17/07/20 04:15:41 INFO spark.SecurityManager: Changing modify acls to: cloudera
17/07/20 04:15:41 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cloudera); users with modify permissions: Set(cloudera)
17/07/20 04:15:41 INFO yarn.Client: Submitting application 8 to ResourceManager
17/07/20 04:15:41 INFO impl.YarnClientImpl: Submitted application application_1500545966176_0008
17/07/20 04:15:42 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:42 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.cloudera
start time: 1500549341177
final status: UNDEFINED
tracking URL: http://quickstart.cloudera:8088/proxy/application_1500545966176_0008/
user: cloudera
17/07/20 04:15:43 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:44 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:45 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:46 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:47 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:48 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:49 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:50 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:51 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:52 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:53 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:54 INFO yarn.Client: Application report for application_1500545966176_0008 (state: ACCEPTED)
17/07/20 04:15:55 INFO yarn.Client: Application report for application_1500545966176_0008 (state: FAILED)
17/07/20 04:15:55 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1500545966176_0008 failed 2 times due to AM Container for appattempt_1500545966176_0008_000002 exited with exitCode: 15
For more detailed output, check application tracking page:http://quickstart.cloudera:8088/proxy/application_1500545966176_0008/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1500545966176_0008_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
at org.apache.hadoop.util.Shell.run(Shell.java:504)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 15
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.cloudera
start time: 1500549341177
final status: FAILED
tracking URL: http://quickstart.cloudera:8088/cluster/app/application_1500545966176_0008
user: cloudera
Exception in thread "main" org.apache.spark.SparkException: Application application_1500545966176_0008 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1030)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1077)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/07/20 04:15:55 INFO util.ShutdownHookManager: Shutdown hook called
17/07/20 04:15:55 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-3a2fec6b-6148-49a9-9db7-7a7cca99c586
[cloudera@quickstart ~]$ clearc^C
[cloudera@quickstart ~]$ ^C
[cloudera@quickstart ~]$ spark-submit --master yarn-cluster /media/sf_SparkS3AirFlow/target/scala-2.11/spark-emr-test.jar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/flume-ng/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/07/20 04:50:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/07/20 04:50:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/07/20 04:50:04 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers
17/07/20 04:50:04 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/07/20 04:50:04 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
17/07/20 04:50:04 INFO yarn.Client: Setting up container launch context for our AM
17/07/20 04:50:04 INFO yarn.Client: Setting up the launch environment for our AM container
17/07/20 04:50:04 INFO yarn.Client: Preparing resources for our AM container
17/07/20 04:50:06 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
17/07/20 04:50:06 INFO yarn.Client: Uploading resource file:/usr/lib/spark/lib/spark-assembly-1.6.0-cdh5.10.0-hadoop2.6.0-cdh5.10.0.jar -> hdfs://quickstart.cloudera:8020/user/cloudera/.sparkStaging/application_1500545966176_0009/spark-assembly-1.6.0-cdh5.10.0-hadoop2.6.0-cdh5.10.0.jar
17/07/20 04:50:07 INFO yarn.Client: Uploading resource file:/media/sf_SparkS3AirFlow/target/scala-2.11/spark-emr-test.jar -> hdfs://quickstart.cloudera:8020/user/cloudera/.sparkStaging/application_1500545966176_0009/spark-emr-test.jar
17/07/20 04:50:09 INFO yarn.Client: Uploading resource file:/tmp/spark-2dbf7a93-4377-4ad1-8e78-56330ee03b7f/__spark_conf__8411937016766228982.zip -> hdfs://quickstart.cloudera:8020/user/cloudera/.sparkStaging/application_1500545966176_0009/__spark_conf__8411937016766228982.zip
17/07/20 04:50:09 WARN hdfs.DFSClient: Caught exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1281)
at java.lang.Thread.join(Thread.java:1355)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder(DFSOutputStream.java:951)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.endBlock(DFSOutputStream.java:689)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:878)
17/07/20 04:50:09 INFO spark.SecurityManager: Changing view acls to: cloudera
17/07/20 04:50:09 INFO spark.SecurityManager: Changing modify acls to: cloudera
17/07/20 04:50:09 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cloudera); users with modify permissions: Set(cloudera)
17/07/20 04:50:09 INFO yarn.Client: Submitting application 9 to ResourceManager
17/07/20 04:50:09 INFO impl.YarnClientImpl: Submitted application application_1500545966176_0009
17/07/20 04:50:10 INFO yarn.Client: Application report for application_1500545966176_0009 (state: ACCEPTED)
17/07/20 04:50:10 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.cloudera
start time: 1500551409216
final status: UNDEFINED
tracking URL: http://quickstart.cloudera:8088/proxy/application_1500545966176_0009/
user: cloudera
17/07/20 04:50:11 INFO yarn.Client: Application report for application_1500545966176_0009 (state: ACCEPTED)
17/07/20 04:50:12 INFO yarn.Client: Application report for application_1500545966176_0009 (state: ACCEPTED)
17/07/20 04:50:13 INFO yarn.Client: Application report for application_1500545966176_0009 (state: ACCEPTED)
17/07/20 04:50:14 INFO yarn.Client: Application report for application_1500545966176_0009 (state: ACCEPTED)
17/07/20 04:50:15 INFO yarn.Client: Application report for application_1500545966176_0009 (state: ACCEPTED)
17/07/20 04:50:16 INFO yarn.Client: Application report for application_1500545966176_0009 (state: ACCEPTED)
17/07/20 04:50:17 INFO yarn.Client: Application report for application_1500545966176_0009 (state: ACCEPTED)
17/07/20 04:50:18 INFO yarn.Client: Application report for application_1500545966176_0009 (state: ACCEPTED)
17/07/20 04:50:19 INFO yarn.Client: Application report for application_1500545966176_0009 (state: ACCEPTED)
17/07/20 04:50:20 INFO yarn.Client: Application report for application_1500545966176_0009 (state: ACCEPTED)
17/07/20 04:50:21 INFO yarn.Client: Application report for application_1500545966176_0009 (state: ACCEPTED)
17/07/20 04:50:22 INFO yarn.Client: Application report for application_1500545966176_0009 (state: FAILED)
17/07/20 04:50:22 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1500545966176_0009 failed 2 times due to AM Container for appattempt_1500545966176_0009_000002 exited with exitCode: 15
For more detailed output, check application tracking page:http://quickstart.cloudera:8088/proxy/application_1500545966176_0009/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1500545966176_0009_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
at org.apache.hadoop.util.Shell.run(Shell.java:504)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 15
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.cloudera
start time: 1500551409216
final status: FAILED
tracking URL: http://quickstart.cloudera:8088/cluster/app/application_1500545966176_0009
user: cloudera
Exception in thread "main" org.apache.spark.SparkException: Application application_1500545966176_0009 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1030)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1077)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/07/20 04:50:22 INFO util.ShutdownHookManager: Shutdown hook called
17/07/20 04:50:22 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-2dbf7a93-4377-4ad1-8e78-56330ee03b7f
[cloudera@quickstart ~]$
Container: container_1500545966176_0011_02_000001 on quickstart.cloudera_44248
================================================================================
LogType:stderr
Log Upload Time:Thu Jul 20 05:02:55 -0700 2017
LogLength:4104
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/flume-ng/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/07/20 05:02:45 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
17/07/20 05:02:49 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1500545966176_0011_000002
17/07/20 05:02:53 INFO spark.SecurityManager: Changing view acls to: yarn,cloudera
17/07/20 05:02:53 INFO spark.SecurityManager: Changing modify acls to: yarn,cloudera
17/07/20 05:02:53 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, cloudera); users with modify permissions: Set(yarn, cloudera)
17/07/20 05:02:53 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
17/07/20 05:02:53 INFO yarn.ApplicationMaster: Waiting for spark context initialization...
17/07/20 05:02:53 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:780)
at org.apache.spark.sql.SparkSession$Builder.master(SparkSession.scala:833)
at Main$.main(Main.scala:8)
at Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552)
17/07/20 05:02:53 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;)
17/07/20 05:02:53 ERROR yarn.ApplicationMaster: Uncaught exception:
java.util.concurrent.ExecutionException: Boxed Error
at scala.concurrent.impl.Promise$.resolver(Promise.scala:55)
at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244)
at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:570)
Caused by: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:780)
at org.apache.spark.sql.SparkSession$Builder.master(SparkSession.scala:833)
at Main$.main(Main.scala:8)
at Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552)
17/07/20 05:02:53 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;)
17/07/20 05:02:53 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1500545966176_0011
17/07/20 05:02:53 INFO util.ShutdownHookManager: Shutdown hook called
LogType:stdout
Log Upload Time:Thu Jul 20 05:02:55 -0700 2017
LogLength:0
Log Contents:
谢谢有什么错误？你能打印它的堆栈跟踪吗？@dumitru我刚刚用堆栈跟踪编辑了这个问题：）你能从你的sbt文件中粘贴吗，看起来你编译jar的scala版本和运行jar的scala版本（cloudera）不是sameCloudera也提供了他们自己的Spark依赖包，您应该使用它来避免依赖冲突。是的，正确使用cloudera存储库来处理spark依赖关系
Container: container_1500545966176_0011_02_000001 on quickstart.cloudera_44248
================================================================================
LogType:stderr
Log Upload Time:Thu Jul 20 05:02:55 -0700 2017
LogLength:4104
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/flume-ng/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/07/20 05:02:45 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
17/07/20 05:02:49 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1500545966176_0011_000002
17/07/20 05:02:53 INFO spark.SecurityManager: Changing view acls to: yarn,cloudera
17/07/20 05:02:53 INFO spark.SecurityManager: Changing modify acls to: yarn,cloudera
17/07/20 05:02:53 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, cloudera); users with modify permissions: Set(yarn, cloudera)
17/07/20 05:02:53 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
17/07/20 05:02:53 INFO yarn.ApplicationMaster: Waiting for spark context initialization...
17/07/20 05:02:53 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:780)
at org.apache.spark.sql.SparkSession$Builder.master(SparkSession.scala:833)
at Main$.main(Main.scala:8)
at Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552)
17/07/20 05:02:53 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;)
17/07/20 05:02:53 ERROR yarn.ApplicationMaster: Uncaught exception:
java.util.concurrent.ExecutionException: Boxed Error
at scala.concurrent.impl.Promise$.resolver(Promise.scala:55)
at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244)
at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:570)
Caused by: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:780)
at org.apache.spark.sql.SparkSession$Builder.master(SparkSession.scala:833)
at Main$.main(Main.scala:8)
at Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552)
17/07/20 05:02:53 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;)
17/07/20 05:02:53 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1500545966176_0011
17/07/20 05:02:53 INFO util.ShutdownHookManager: Shutdown hook called
LogType:stdout
Log Upload Time:Thu Jul 20 05:02:55 -0700 2017
LogLength:0
Log Contents: