Scala/Spark:由以下原因引起：java.lang.ClassNotFoundException:org.apache.Spark.Logging

我对scala和spark还很陌生。试图修复spark/scala开发的设置。我被版本和丢失的罐子弄糊涂了。我搜索了stackoverflow，但仍然停留在这个问题上。可能是缺少什么或配置错误

运行命令：

me@Mycomputer:~/spark-2.1.0$ bin/spark-submit --class ETLApp /home/me/src/etl/target/scala-2.10/etl-assembly-0.1.0.jar

输出：

...
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
...
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging

build.sbt：

name := "etl"

version := "0.1.0"
scalaVersion := "2.10.5"
javacOptions ++= Seq("-source", "1.8", "-target", "1.8")
mainClass := Some("ETLApp")

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.2" % "provided";
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.2" % "provided";
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.5.2" % "provided";
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" % "1.5.2";
libraryDependencies += "com.datastax.spark"  %% "spark-cassandra-connector" % "1.5.0-M2";
libraryDependencies += "org.apache.curator" % "curator-recipes" % "2.6.0"
libraryDependencies += "org.apache.curator" % "curator-test" % "2.6.0"
libraryDependencies += "args4j" % "args4j" % "2.32"

java版本 java版本“1.8.0_101”

scala版本 2.10.5

星火版 2.1.0

---

欢迎任何提示。谢谢

在这种情况下，提交给spark时，您的jar必须携带所有依赖的类

---

在maven中，这可以通过程序集插件和带有依赖项描述符的jar实现。有了sbt，谷歌很快发现了这一点：

您可以按如下方式更改build.sbt：

name := "etl"

version := "0.1.0"

scalaVersion := "2.10.5"

scalacOptions ++= Seq("-deprecation",
  "-feature",
  "-Xfuture",
  "-encoding",
  "UTF-8",
  "-unchecked",
  "-language:postfixOps")

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.2" % Provided,
"org.apache.spark" %% "spark-sql" % "1.5.2" % Provided,
"org.apache.spark" %% "spark-streaming" % "1.5.2" %  Provided,
"org.apache.spark" %% "spark-streaming-kafka" % "1.5.2" % Provided,
"com.datastax.spark"  %% "spark-cassandra-connector" % "1.5.0-M2",
"org.apache.curator" % "curator-recipes" % "2.6.0",
"org.apache.curator" % "curator-test" % "2.6.0",
"args4j" % "args4j" % "2.32")

mainClass in assembly := Some("your.package.name.ETLApp")

assemblyJarName in assembly := s"${name.value}-${version.value}.jar"

assemblyMergeStrategy in assembly := {
  case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
  case m if m.toLowerCase.matches("meta-inf.*\\.sf$") => MergeStrategy.discard
  case "reference.conf" => MergeStrategy.concat
  case x: String if x.contains("UnusedStubClass.class") => MergeStrategy.first
  case _ => MergeStrategy.first
}

将sbt程序集插件添加到项目根目录中项目目录下的plugins.sbt文件中。在项目根目录中的终端（Linux）或CMD（Windows）中运行

sbt assembly

，将为您下载所有依赖项并创建一个U