Serialization Spark com.fasterxml.jackson.module错误
这有点奇怪。运行非常简单的Serialization Spark com.fasterxml.jackson.module错误,serialization,apache-spark,jackson,Serialization,Apache Spark,Jackson,这有点奇怪。运行非常简单的sparkContext.parallelize(列表(“1”、“2”、“3”))时 我得到以下错误: java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/faster
sparkContext.parallelize(列表(“1”、“2”、“3”))时
我得到以下错误:
java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;
我猜在一些库的依赖关系中存在一些冲突。我的build.sbt如下所示:
scalaVersion := "2.11.7"
//Library repositories
resolvers ++= Seq(
Resolver.mavenLocal,
"Scala-Tools Maven2 Repository" at "http://scala-tools.org/repo-releases",
"Java.net repository" at "http://download.java.net/maven/2",
"GeoTools" at "http://download.osgeo.org/webdav/geotools",
"Apache" at "https://repository.apache.org/service/local/repositories/releases/content",
"Cloudera" at "https://repository.cloudera.com/artifactory/cloudera-repos/",
"OpenGeo Maven Repository" at "http://repo.opengeo.org",
"Typesafe" at "https://repo.typesafe.com/typesafe/releases/",
"Spray Repository" at "http://repo.spray.io"
)
//Library versions
val geotools_version = "13.2"
val accumulo_version = "1.6.0-cdh5.1.4"
val hadoop_version = "2.6.0-cdh5.4.5"
val hadoop_client_version = "2.6.0-mr1-cdh5.4.5"
val geowave_version = "0.9.0-SNAPSHOT"
val akka_version = "2.4.0"
val spray_version = "1.3.3"
val spark_version = "1.5.0"
//Library Dependencies
libraryDependencies ++= Seq(
//Scala
"org.scala-lang" % "scala-library" % scalaVersion.value,
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
//GeoTools
"org.geotools" % "gt-data" % geotools_version,
"org.geotools" % "gt-geojson" % geotools_version,
//Apache
"org.apache.accumulo" % "accumulo-core" % accumulo_version,
"org.apache.hadoop" % "hadoop-common" % hadoop_version,
"org.apache.hadoop" % "hadoop-client" % hadoop_client_version,
//Geowave
"mil.nga.giat" % "geowave-core-store" % geowave_version,
"mil.nga.giat" % "geowave-datastore-accumulo" % geowave_version,
"mil.nga.giat" % "geowave-adapter-vector" % geowave_version,
//Other
"com.typesafe" % "config" % "1.3.0",
//Spray - Akka
"com.typesafe.akka" %% "akka-actor" % akka_version,
"io.spray" %% "spray-can" % spray_version,
"io.spray" %% "spray-routing" % spray_version,
"io.spray" %% "spray-testkit" % spray_version % "test",
//Spark
"org.apache.spark" %% "spark-core" % spark_version,
"com.typesafe.play" %% "play-json" % "2.5.0-M1",
//Testing
"org.scalatest" % "scalatest_2.11" % "2.2.4" % "test"
).map(
_.excludeAll(ExclusionRule(organization = "org.mortbay.jetty"))
)
test in assembly := {}
有没有关于去哪里找的建议
感谢实际上,这是一个依赖冲突,因为json播放和spark使用了不同版本的jackson数据绑定。此
build.sbt
似乎解决了以下问题:
scalaVersion := "2.11.7"
//Library repositories
resolvers ++= Seq(
Resolver.mavenLocal,
"Scala-Tools Maven2 Repository" at "http://scala-tools.org/repo-releases",
"Java.net repository" at "http://download.java.net/maven/2",
"GeoTools" at "http://download.osgeo.org/webdav/geotools",
"Apache" at "https://repository.apache.org/service/local/repositories/releases/content",
"Cloudera" at "https://repository.cloudera.com/artifactory/cloudera-repos/",
"OpenGeo Maven Repository" at "http://repo.opengeo.org",
"Typesafe" at "https://repo.typesafe.com/typesafe/releases/",
"Spray Repository" at "http://repo.spray.io"
)
//Library versions
val geotools_version = "13.2"
val accumulo_version = "1.6.0-cdh5.1.4"
val hadoop_version = "2.6.0-cdh5.4.5"
val hadoop_client_version = "2.6.0-mr1-cdh5.4.5"
val geowave_version = "0.9.0-SNAPSHOT"
val akka_version = "2.4.0"
val spray_version = "1.3.3"
val spark_version = "1.5.2"
//Library Dependencies
libraryDependencies ++= Seq(
//Scala
"org.scala-lang" % "scala-library" % scalaVersion.value,
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
//GeoTools
"org.geotools" % "gt-data" % geotools_version,
"org.geotools" % "gt-geojson" % geotools_version,
/** ********************************************** PROVIDED ****************************************/
// Apache
// "org.apache.accumulo" % "accumulo-core" % accumulo_version % "provided",
// "org.apache.hadoop" % "hadoop-common" % hadoop_version% "provided",
// "org.apache.hadoop" % "hadoop-client" % hadoop_client_version% "provided",
//
// //Geowave
// "mil.nga.giat" % "geowave-core-store" % geowave_version % "provided",
// "mil.nga.giat" % "geowave-datastore-accumulo" % geowave_version % "provided",
// "mil.nga.giat" % "geowave-adapter-vector" % geowave_version % "provided",
/** **********************************************************************************************/
//Apache
"org.apache.accumulo" % "accumulo-core" % accumulo_version,
"org.apache.hadoop" % "hadoop-common" % hadoop_version,
"org.apache.hadoop" % "hadoop-client" % hadoop_client_version,
//Geowave
"mil.nga.giat" % "geowave-core-store" % geowave_version,
"mil.nga.giat" % "geowave-datastore-accumulo" % geowave_version,
"mil.nga.giat" % "geowave-adapter-vector" % geowave_version,
//Other
"com.typesafe" % "config" % "1.3.0",
//Spray - Akka
"com.typesafe.akka" %% "akka-actor" % akka_version,
"io.spray" %% "spray-can" % spray_version,
"io.spray" %% "spray-routing" % spray_version,
"io.spray" %% "spray-testkit" % spray_version % "test",
"com.typesafe.play" %% "play-json" % "2.5.0-M1"
exclude("com.fasterxml.jackson.core", "jackson-databind"),
//Spark
"org.apache.spark" %% "spark-core" % spark_version,
//Testing
"org.scalatest" % "scalatest_2.11" % "2.2.4" % "test"
).map(
_.excludeAll(
ExclusionRule(organization = "org.mortbay.jetty")
)
)
test in assembly := {}
在相同的项目中使用Spring Boot和Spark可能会遇到相同的问题。以防万一-从Spark以外的其他项目中排除jackson数据绑定依赖项。Spring Boot 1.3.1+Apache Spark 1.6.0问题通过添加依赖项得到解决:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.4.4</version>
</dependency>
com.fasterxml.jackson.core
杰克逊数据绑定
2.4.4
不排除任何其他所需的依赖项。只需在SBT中添加这些行即可
dependencyOverrides ++= Set(
"com.fasterxml.jackson.core" % "jackson-databind" % "2.4.4"
)
我也有同样的问题,NJNEX解决方案对我有效!我使用SpringBoot1.3.3+Spark1.6.1和Gradle2.9作为构建工具。以下是Gradle用户的解决方案:
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.4.4'
如果您遇到问题,可以随时尝试您可以找到的其他版本(适用于Maven和SBT用户)如果您有jackson core asl和jackson mapper asl,请删除它们。这就解决了我的类似问题。如果有人使用Maven进行构建管理,这是与spark version兼容的版本
2.11
。使用最高2.9.0版本的jackson
的所有版本进行测试。只有这种组合起作用
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>foreverLearn</groupId>
<artifactId>ClusteringIndustry</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<jackson.version>2.6.0</jackson.version>
<spark.version>2.3.0</spark.version>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-mllib -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>${spark.version}</version>
<!-- <scope>runtime</scope> -->
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>2.7.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.660</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.6</version>
</dependency>
</dependencies>
</project>
有没有机会用maven而不是sbt来解决这个问题?我不认为这会有什么不同。在这方面,依赖解决方案非常相似。是的,我只是不熟悉sbt,它是allI使用的Spring Boot v1.3.1和Spark v1.6.0,以及关于
com.fasterxml.jackson.core:jackson数据绑定:2.6.4
的冲突。显然,org.springframework.boot:springbootstarterweb:1.3.1.RELEASE
需要v2.6.4,Spark需要v2.4.4。我必须做的是排除v2.6.4,然后明确地依赖于v.2.4.4。目前似乎正在工作。+1它工作:-)。您可能希望为org.codehaus.jackson:jackson mapper asl
输入并排除,因为类路径中现在有两个版本的AsArraySerializerBase
,谢谢您解决了我的问题+1.我也在answers中发布了gradle等价物,如果有人需要的话。这不适用于我的Spark 1.6.1和play json 2.4.8,请尝试更改jackson databind的版本。将其设置为与您的play json版本相同。(示例:当前您使用的是2.4.4,因此在您的情况下将其更改为2.4.8。)
name := "coursera"
version := "0.1"
scalaVersion := "2.12.10"
val sparkVersion = "3.0.0-preview2"
val playVersion="2.8.1"
val jacksonVersion="2.10.1"
//override if you wish to
//dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % jacksonVersion
//dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % jacksonVersion
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-streaming" % sparkVersion,
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"com.typesafe.play" %% "play-json" % playVersion
)