Scala java.util.concurrent.ExecutionException:java.lang.NoSuchMethodError运行kinesis spark作业时出现异常

我很难找出spark（2.1.0）工作与scala的依赖关系

我的build.sbt文件：

name := "test"

version := "0.0.1"

scalaVersion := "2.11.0"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.1.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-streaming-kinesis-asl" % "2.1.0"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.5.1"

assemblyJarName in assembly := "test.jar"


mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
    case m if m.startsWith("META-INF") => MergeStrategy.discard
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first
    case PathList("org", "apache", xs @ _*) => MergeStrategy.first
    case PathList("org", "jboss", xs @ _*) => MergeStrategy.first
    case "about.html"  => MergeStrategy.rename
    case "reference.conf" => MergeStrategy.concat
    case _ => MergeStrategy.first
  }
}

exportJars:= true

mainClass in assembly := Some("test.Job")
```

---

你在aws emr上运行这个吗？ 如果是这样，以下链接可能会有所帮助：

---

本质上，aws EMR支持protobuf 2.5，但是spark core、spark streaming和spark streaming kinesis asl版本都依赖于protobuf 2.6.1，当我们遇到这个问题时，我们解决这个问题的方法是通过一个带阴影的罐子，上面的两个链接给出了如何设置的好例子

您是否在aws emr上运行此功能？ 如果是这样，以下链接可能会有所帮助：

本质上，aws EMR支持protobuf 2.5，但是spark core、spark streaming和spark streaming kinesis asl版本都依赖于protobuf 2.6.1，当我们遇到这个问题时，我们解决这个问题的方法是通过一个带阴影的罐子，上面的两个链接给出了如何设置的好例子

你查过了吗？你查过了吗？

17/03/08 05:19:15 INFO storage.BlockManager: Removing RDD 87
17/03/08 05:19:15 INFO storage.BlockManager: Removing RDD 86
17/03/08 05:19:15 INFO storage.BlockManager: Removing RDD 85
17/03/08 05:19:15 ERROR worker.Worker: Worker.run caught exception, sleeping for 1000 milli seconds!
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.amazonaws.services.kinesis.model.GetRecordsResult.getMillisBehindLatest()Ljava/lang/Long;
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.checkAndSubmitNextTask(ShardConsumer.java:156)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.consumeShard(ShardConsumer.java:125)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:335)
    at org.apache.spark.streaming.kinesis.KinesisReceiver$$anon$1.run(KinesisReceiver.scala:174)
Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.amazonaws.services.kinesis.model.GetRecordsResult.getMillisBehindLatest()Ljava/lang/Long;
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:192)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.checkAndSubmitNextTask(ShardConsumer.java:136)
    ... 3 more
Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.kinesis.model.GetRecordsResult.getMillisBehindLatest()Ljava/lang/Long;
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.getRecordsResultAndRecordMillisBehindLatest(ProcessTask.java:291)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.getRecordsResult(ProcessTask.java:249)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:120)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)