SCALA和弹性搜索:向类路径(Databricks)添加符号
:) 我遇到了一个我不知道如何修复的错误。。我真的找不到这个SchemaRDD类型的优秀文档以及如何使用它 build.sbt包含:SCALA和弹性搜索:向类路径(Databricks)添加符号,scala,
elasticsearch,dependencies,sbt,databricks,Scala,
elasticsearch,Dependencies,Sbt,Databricks,:) 我遇到了一个我不知道如何修复的错误。。我真的找不到这个SchemaRDD类型的优秀文档以及如何使用它 build.sbt包含: scalaVersion := "2.11.12" libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.0" libraryDependencies += "org.scalaj" %% "scalaj-http" % "2.4.1" libraryDependencies += "io
scalaVersion := "2.11.12"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.0"
libraryDependencies += "org.scalaj" %% "scalaj-http" % "2.4.1"
libraryDependencies += "io.spray" %% "spray-json" % "1.3.5"
libraryDependencies += "com.amazonaws" % "aws-java-sdk-core" % "1.11.534"
libraryDependencies += "com.amazonaws" % "aws-encryption-sdk-java" % "1.3.6"
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.11.550"
libraryDependencies += "com.typesafe" % "config" % "1.3.4"
libraryDependencies += "org.elasticsearch" %% "elasticsearch-spark-1.2" % "2.4.4"
Symbol 'type org.apache.spark.sql.SchemaRDD' is missing from the classpath.
[error] This symbol is required by 'value org.elasticsearch.spark.sql.package.rdd'.
[error] Make sure that type SchemaRDD is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
[error] A full rebuild may help if 'package.class' was compiled against an incompatible version of org.apache.spark.sql.
错误:
scalaVersion := "2.11.12"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.0"
libraryDependencies += "org.scalaj" %% "scalaj-http" % "2.4.1"
libraryDependencies += "io.spray" %% "spray-json" % "1.3.5"
libraryDependencies += "com.amazonaws" % "aws-java-sdk-core" % "1.11.534"
libraryDependencies += "com.amazonaws" % "aws-encryption-sdk-java" % "1.3.6"
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.11.550"
libraryDependencies += "com.typesafe" % "config" % "1.3.4"
libraryDependencies += "org.elasticsearch" %% "elasticsearch-spark-1.2" % "2.4.4"
Symbol 'type org.apache.spark.sql.SchemaRDD' is missing from the classpath.
[error] This symbol is required by 'value org.elasticsearch.spark.sql.package.rdd'.
[error] Make sure that type SchemaRDD is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
[error] A full rebuild may help if 'package.class' was compiled against an incompatible version of org.apache.spark.sql.
非常感谢您的支持!:) Dependency
elasticsearch-spark-1.2
适用于spark 1.x,需要使用elasticsearch-spark-20
。是为Spark 2.3构建的
libraryDependencies += "org.elasticsearch" %% "elasticsearch-spark-20" % "7.1.1"
谢谢,最后我发现我使用了一个错误的库,我不得不使用elasticsearch hadoop。但是非常感谢您的回复!!