Scala 如何将Jackson依赖项与AWS Glue一起使用
我尝试在AWS胶水作业中使用库。我使用它来创建包含所有依赖项的jar 在包括杰克逊在内的情况下Scala 如何将Jackson依赖项与AWS Glue一起使用,scala,jackson,aws-glue,Scala,Jackson,Aws Glue,我尝试在AWS胶水作业中使用库。我使用它来创建包含所有依赖项的jar 在包括杰克逊在内的情况下 dependencyOverrides ++= Seq( "com.fasterxml.jackson.core" % "jackson-core" % "2.9.3", "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.3", "com.fasterxml.jackson.core" % "jackson-annotatio
dependencyOverrides ++= Seq(
"com.fasterxml.jackson.core" % "jackson-core" % "2.9.3",
"com.fasterxml.jackson.core" % "jackson-databind" % "2.9.3",
"com.fasterxml.jackson.core" % "jackson-annotations" % "2.9.3",
)
错误是
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.3
ERROR ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ArrayNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/List;)V
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ArrayNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/List;)V
完整日志
如果jackson被排除在外
excludeDependencies ++= Seq(
ExclusionRule("com.fasterxml.jackson.core")
)
错误是
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.3
ERROR ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ArrayNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/List;)V
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ArrayNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/List;)V
错误应用程序管理员:用户类引发异常:java.lang.NoSuchMethodError:com.fasterxml.jackson.databind.node.ArrayNode。(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/List;)V
java.lang.NoSuchMethodError:com.fasterxml.jackson.databind.node.ArrayNode(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/List;)V
完整日志AWS Glue带来了它对Jackson的依赖性,但我不确定是哪个版本。您可以检查glue-assembly.jar内容- 问题在于,您的程序集和粘合程序集最终都位于Spark executor上的同一类路径上 在使用Google Big Query Connector(主要是jackson和guava)时,我在我的项目中遇到了同样的不兼容库问题,我通过在我的组装jar中着色这些库来解决这个问题,如下所示:
lazy val root = (project in file(".")).
settings(
...,
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.fasterxml.**" -> "my.shaded.@0").inAll,
// other shade rules
),
libraryDependencies ++= ...
)