Scala sbt程序集：合并错误-重复数据消除

我使用

sbt assembly

获得这些错误

我使用的火花似乎是这个问题的根源

val Spark =  Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion
)

错误：

[error] 12 errors were encountered during merge
[trace] Stack trace suppressed: run last coreBackend/*:assembly for the full output.
[trace] Stack trace suppressed: run last core/*:assembly for the full output.
[trace] Stack trace suppressed: run last commons/*:assembly for the full output.
[error] (coreBackend/*:assembly) deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.osgi/org.osgi.core/org.osgi.core-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.osgi/org.osgi.compendium/org.osgi.compendium-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Absent.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Absent.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Function.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Function.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Optional$1$1.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1$1.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Optional$1.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Optional.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Present.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Present.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Supplier.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Supplier.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-core_2.11/spark-core_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-launcher_2.11/spark-launcher_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.spark-project.spark/unused/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-shuffle_2.11/spark-network-shuffle_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-unsafe_2.11/spark-unsafe_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-sql_2.11/spark-sql_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-catalyst_2.11/spark-catalyst_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-streaming_2.11/spark-streaming_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] (core/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.core/jars/org.osgi.core-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.compendium/jars/org.osgi.compendium-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Absent.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Absent.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Function.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Function.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Optional$1$1.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1$1.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Optional$1.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Optional.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Present.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Present.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Supplier.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Supplier.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-core_2.11/jars/spark-core_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-launcher_2.11/jars/spark-launcher_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.spark-project.spark/unused/jars/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-shuffle_2.11/jars/spark-network-shuffle_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-unsafe_2.11/jars/spark-unsafe_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-sql_2.11/jars/spark-sql_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-catalyst_2.11/jars/spark-catalyst_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-streaming_2.11/jars/spark-streaming_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] (commons/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.core/jars/org.osgi.core-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.compendium/jars/org.osgi.compendium-4.3.1.jar:OSGI-OPT/bnd.bnd

我在这里尝试了所有推荐的解决方案，但没有成功

---

这不是问题的确切答案，但它是一种解决方法

我希望这能节省几百个工时

使用而不是

sbt组件

添加到

插件.sbt

：

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.0.0")

在您的

build.sbt

enablePlugins(JavaAppPackaging)
enablePlugins(UniversalPlugin)

要为多个Scala版本生成文件，请使用+

+ universal:packageBin

输出将告诉您文件的创建位置

不幸的是，生成的jar是压缩的。这不是一个肥罐子。（要生成fat jar，需要

sbt程序集

，它也有同样的问题）

为了克服这个问题，我制作了一个简单的脚本（在SBT中），它可以解压生成的文件并将jar路径写入一个文件，这样我就可以轻松地构建Spark submit脚本

packageBin in TxtFormat := {

    val zippedJar = "core-backend-1.0.zip"
    val basePath = target.value / "universal"

    // Unzip to folder of JARs
    IO.unzip(basePath / zippedJar, basePath)

    val fileMappings = (mappings in Universal).value
    val sparkScriptOut = basePath / s"${packageName.value}.txt"

    // append all mappings to the list
    fileMappings foreach {
        case (file, name) => IO.append(sparkScriptOut, s"core-backend-1.0/$name${IO.Newline}")
    }
    sparkScriptOut
}

构建zip后，使用以下命令执行任务：

+ txtFormat:packageBin

---

这对我很有用。

Spark依赖项应该由集群提供，添加“提供”：

---

我知道这是一个老生常谈的问题，但迄今为止给出的任何解决方案都不能解决我遇到的问题

似乎在某些情况下，未“提供”的jar依赖于已提供的jar。如果可传递依赖项拉入内容冲突的JAR，您可以看到相同的错误。在某些情况下，将这些JAR添加为标记为“provided”的依赖项并不能解决构建程序集时它们之间的冲突。我找到的唯一解决方案是如下所示将它们从程序集中显式排除

assemblyExcludedJars in assembly := {
    val cp = (fullClasspath in assembly).value
    cp filter { el =>
        (el.data.getName == "unused-1.0.0.jar") ||
        (el.data.getName == "spark-tags_2.11-2.1.0.jar")  
    }
}

---

自Spark 1.4.1以来，我也遇到了同样的问题，我尝试了所有推荐的解决方案，但没有一个有效。后来我不得不改用maven。@eliasah我想出了一个解决办法——当我们有更好的工具时，仍然使用SBT作为maven几乎是不可能的。我抛弃了

sbt assembly

，因为相同的错误正在许多不同的库中出现，而且解决方法充其量也是有问题的，这（对我来说）表明了未来项目的问题。使用sbt本机打包机。我会把它作为一个答案发布，但在上下文中，这只是一个解决办法。

val Spark =  Seq(
    "org.apache.spark" %% "spark-core" % sparkVersion % Provided,
    "org.apache.spark" %% "spark-sql" % sparkVersion % Provided,
    "org.apache.spark" %% "spark-streaming" % sparkVersion % Provided
)

assemblyExcludedJars in assembly := {
    val cp = (fullClasspath in assembly).value
    cp filter { el =>
        (el.data.getName == "unused-1.0.0.jar") ||
        (el.data.getName == "spark-tags_2.11-2.1.0.jar")  
    }
}