Scala 无法编译Spark流示例：getting updateStateByKey不是org.apache.Spark.Streaming.dstream.dstream错误的成员

我想更好地理解Spark Streaming，所以我将StatefulNetworkWordCount.scala示例复制到它自己的目录中，并按原样粘贴，创建了一个简单的sbt配置文件，并尝试进行编译，但它似乎不起作用。它为什么抱怨API？ 有什么想法吗？最初的示例确实可以很好地编译，但我想更改它，而不必像默认设置那样编译整个spark目录

谢谢！ 马特

错误：

urbanlegends-2:streamingtest mlieber$sbt package
[info] Set current project to Simple Project (in build file:/Users/mlieber/app/spark2/spark/examples/lieber/streamingtest/)
[info] Compiling 2 Scala sources to /Users/mlieber/app/spark2/spark/examples/lieber/streamingtest/target/scala-2.10/classes...
[error] /Users/mlieber/app/spark2/spark/examples/lieber/streamingtest/src/main/scala/com/cloudera/streamingtest/StatefulNetworkWordCount2.scala:58: value updateStateByKey is not a member of org.apache.spark.streaming.dstream.DStream[(String, Int)]
[error]     **val stateDstream = wordDstream.updateStateByKey[Int](newUpdateFunc,
[error]**                                    ^
[error] one error found
[error] (compile:compile) Compilation failed
[error] Total time: 3 s, completed Mar 16, 2015 4:23:15 PM
urbanlegends-2:streamingtest mlieber$ 

---

看起来问题出在库版本不可用。尝试将

“spark core”%%“1.2.1”

和

“spark-streaming_2.10”%%“1.2.1”

替换为

"spark-core" % "1.2.0"

"spark-streaming_2.10" % "1.2.0"

[更新] 或

---

您还需要添加导入org.apache.spark.streaming.StreamingContext。

我通过将spark版本更改为1.3.0解决了这个问题。 请参阅我的sbt文件：

$ more build.sbt
import AssemblyKeys._ // put this at the top of the file

name := "Tutorial"

scalaVersion := "2.10.3"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-streaming" % "1.3.0",
  "org.apache.spark" %% "spark-streaming-twitter" % "1.3.0"
)

resourceDirectory in Compile := baseDirectory.value / "resources"

assemblySettings

mergeStrategy in assembly := {
  case m if m.toLowerCase.endsWith("manifest.mf")          => MergeStrategy.discard
  case m if m.toLowerCase.matches("meta-inf.*\\.sf$")      => MergeStrategy.discard
  case "log4j.properties"                                  => MergeStrategy.discard
  case m if m.toLowerCase.startsWith("meta-inf/services/") => MergeStrategy.filterDistinctLines
  case "reference.conf"                                    => MergeStrategy.concat
  case _                                                   => MergeStrategy.first
}

---

我在使用spark 1.2.0时也遇到了同样的错误。存在一些库兼容性问题。在解决此问题之前，只需尝试升级到1.3.0即可解决此问题

scalaVersion := "2.10.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.3.0" % "provided"

---

我已经更新了答案，也许这篇文章可以给你一个合适的例子谢谢-但是更多的错误接踵而至：[error]/Users/mlieber/app/spark2/spark/examples/lieber/streamingtest/src/main/scala/com/cloudera/streamingtest/StatefulNetworkWordCount2.scala:59：重载方法值updateStateByKey和可选项：[]（updateFunc:Iterator[（String，Seq[Int]，Option[Int]）]=>Iterator[（String，Int）]，分区器：证据$2:org.apache.spark.rdd.rdd[（String，Int）]）[error]val stateDstream=wordDstream.updateStateByKey[Int]（newUpdateFunc，[error]^我首先想到的是，根本原因是updateFunc，但由于缺乏信息，很难说

scalaVersion := "2.10.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.3.0" % "provided"