Scala Resolve spark avro error=未能为数据源com.databricks.spark.avro加载类

我正在尝试使用库来处理avro文件。我正在使用SBT：

build.sbt：

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % "1.3.0",
  "com.databricks" %% "spark-avro" % "1.0.0")

tester.scala：

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.sql._
import com.databricks.spark.avro._

object tester {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("SimpleApplication").setMaster("local")
    val sc = new SparkContext(conf)

    // Creates a DataFrame from a specified file
    val df = sqlContext.load("episodes.avro", "com.databricks.spark.avro")
  }
}

在IntelliJ IDE中运行tester时，我得到以下堆栈跟踪：

Exception in thread "main" java.lang.NoClassDefFoundError:        org/apache/avro/mapred/FsInput
    at com.databricks.spark.avro.AvroRelation.newReader(AvroRelation.scala:111)
    at com.databricks.spark.avro.AvroRelation.<init>(AvroRelation.scala:53)
    at com.databricks.spark.avro.DefaultSource.createRelation(DefaultSource.scala:41)
    at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:290)

Exception in thread "main" java.lang.RuntimeException: Failed to load class for data source: com.databricks.spark.avro
    at scala.sys.package$.error(package.scala:27)
    at org.apache.spark.sql.sources.ResolvedDataSource$.lookupDataSource(ddl.scala:194)
    at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:205)
    at org.apache.spark.sql.SQLContext.load(SQLContext.scala:697)

我得到以下堆栈跟踪：

Exception in thread "main" java.lang.NoClassDefFoundError:        org/apache/avro/mapred/FsInput
    at com.databricks.spark.avro.AvroRelation.newReader(AvroRelation.scala:111)
    at com.databricks.spark.avro.AvroRelation.<init>(AvroRelation.scala:53)
    at com.databricks.spark.avro.DefaultSource.createRelation(DefaultSource.scala:41)
    at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:290)

Exception in thread "main" java.lang.RuntimeException: Failed to load class for data source: com.databricks.spark.avro
    at scala.sys.package$.error(package.scala:27)
    at org.apache.spark.sql.sources.ResolvedDataSource$.lookupDataSource(ddl.scala:194)
    at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:205)
    at org.apache.spark.sql.SQLContext.load(SQLContext.scala:697)

---

如何解决此错误？非常感谢您的帮助。谢谢

sbt包将不包含您的依赖项，请尝试替换。

我将build.sbt文件更改为：

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % "1.3.0",
  "com.databricks" %% "spark-avro" % "1.0.0",
  "org.apache.avro" % "avro" % "1.7.7",
  "org.apache.avro" % "avro-mapred" % "1.7.7")

assemblyMergeStrategy in assembly := {
  case PathList("org", "slf4j", xs@_*) => MergeStrategy.first
  case PathList("org", "apache", "spark", xs @_*) => MergeStrategy.first
  case PathList("com", "esotericsoftware", "minlog", xs @_*) => MergeStrategy.first
  case PathList("javax", "activation", xs @_*) => MergeStrategy.first
  case PathList("javax", "servlet", xs @_*) => MergeStrategy.first
  case PathList("javax", "xml", "stream", xs @_*) => MergeStrategy.first
  case PathList("org", "apache", "commons", xs @_*) => MergeStrategy.first
  case PathList("com", "google", "common", xs @_*) => MergeStrategy.first
  case "org/apache/hadoop/yarn/factories/package-info.class" => MergeStrategy.first
  case "org/apache/hadoop/yarn/factory/providers/package-info.class" => MergeStrategy.first
  case "org/apache/hadoop/yarn/util/package-info.class" => MergeStrategy.first
  case x if x.startsWith("META-INF") => MergeStrategy.discard
  case x if x.startsWith("plugin.properties") => MergeStrategy.discard
  case x => {
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
  }
}

并使用命令

$ sbt assembly

来建造这个罐子。现在一切都好了