Java Spark提交失败，IterableCodecProvider的ClassNotFoundException_Java_Mongodb_Maven_Apache Spark

Java Spark提交失败，IterableCodecProvider的ClassNotFoundException

java mongodb maven apache-spark

Java Spark提交失败，IterableCodecProvider的ClassNotFoundException,java,mongodb,maven,apache-spark,Java,Mongodb,Maven,Apache Spark,我正在尝试使用Spark submit在Spark集群上提交一个.jar应用程序。使用的二进制文件是 spark-2.1.1-bin-hadoop2.7 提交时出现的问题是申请失败，例外情况如下： $:/path-to-spark-binary/spark-submit --class Main --master local[*] app.jar Exception in thread "main" java.lang.NoClassDefFoundError: org/bson/codecs

我正在尝试使用Spark submit在Spark集群上提交一个.jar应用程序。使用的二进制文件是 spark-2.1.1-bin-hadoop2.7

提交时出现的问题是申请失败，例外情况如下：

$:/path-to-spark-binary/spark-submit --class Main --master local[*] app.jar

Exception in thread "main" java.lang.NoClassDefFoundError: org/bson/codecs/IterableCodecProvider
at com.mongodb.MongoClient.<clinit>(MongoClient.java:85)
at com.mongodb.MongoClientOptions$Builder.<init>(MongoClientOptions.java:829)
at com.mongodb.MongoClientURI.<init>(MongoClientURI.java:183)

try (SparkSession spark = SparkSession
                    .builder()
                    .config("spark.master", "local")
                    .config("spark.mongodb.input.uri", Constants.sparkMongoDatabaseURI)
                    .config("spark.mongodb.output.uri", Constants.sparkMongoDatabaseURI)
                    .appName("Spark Session 1").getOrCreate()) {

                JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext());

                MongoClientURI mongoClientURI = new MongoClientURI(Constants.dbMongoURI);
                MongoClient mongoClient = new MongoClient(mongoClientURI);

                ...
}

$:/path-to-spark-binary/spark-shell --packages org.mongodb.spark:mongo-spark-connector_2.01:2.0.0

import org.apache.spark.{SparkContext, SparkConf}
import com.mongodb.spark.MongoSpark;

sc.stop()

val sc2 = new SparkContext(new SparkConf().setAppName("shell").set("spark.mongodb.input.uri", "mongodb://my_host:my_port/my_database.my_collection"))

MongoSpark.load(sc2)

当我用相同的Spark二进制文件在我的个人计算机上本地提交时，它也能正常运行

我发现应用程序在创建新的IterableCodecProvider对象时不会失败，但只有在创建新的MongoClient对象时才会失败。这是静态初始化问题吗？在没有上述库的情况下，是否有其他方法与MongoDB数据库进行接口

编辑1：生成异常的代码：

$:/path-to-spark-binary/spark-submit --class Main --master local[*] app.jar

Exception in thread "main" java.lang.NoClassDefFoundError: org/bson/codecs/IterableCodecProvider
at com.mongodb.MongoClient.<clinit>(MongoClient.java:85)
at com.mongodb.MongoClientOptions$Builder.<init>(MongoClientOptions.java:829)
at com.mongodb.MongoClientURI.<init>(MongoClientURI.java:183)

try (SparkSession spark = SparkSession
                    .builder()
                    .config("spark.master", "local")
                    .config("spark.mongodb.input.uri", Constants.sparkMongoDatabaseURI)
                    .config("spark.mongodb.output.uri", Constants.sparkMongoDatabaseURI)
                    .appName("Spark Session 1").getOrCreate()) {

                JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext());

                MongoClientURI mongoClientURI = new MongoClientURI(Constants.dbMongoURI);
                MongoClient mongoClient = new MongoClient(mongoClientURI);

                ...
}

$:/path-to-spark-binary/spark-shell --packages org.mongodb.spark:mongo-spark-connector_2.01:2.0.0

import org.apache.spark.{SparkContext, SparkConf}
import com.mongodb.spark.MongoSpark;

sc.stop()

val sc2 = new SparkContext(new SparkConf().setAppName("shell").set("spark.mongodb.input.uri", "mongodb://my_host:my_port/my_database.my_collection"))

MongoSpark.load(sc2)

编辑2：我也尝试过使用spark shell，但这些相同的命令会生成完全相同的异常：

$:/path-to-spark-binary/spark-submit --class Main --master local[*] app.jar

Exception in thread "main" java.lang.NoClassDefFoundError: org/bson/codecs/IterableCodecProvider
at com.mongodb.MongoClient.<clinit>(MongoClient.java:85)
at com.mongodb.MongoClientOptions$Builder.<init>(MongoClientOptions.java:829)
at com.mongodb.MongoClientURI.<init>(MongoClientURI.java:183)

try (SparkSession spark = SparkSession
                    .builder()
                    .config("spark.master", "local")
                    .config("spark.mongodb.input.uri", Constants.sparkMongoDatabaseURI)
                    .config("spark.mongodb.output.uri", Constants.sparkMongoDatabaseURI)
                    .appName("Spark Session 1").getOrCreate()) {

                JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext());

                MongoClientURI mongoClientURI = new MongoClientURI(Constants.dbMongoURI);
                MongoClient mongoClient = new MongoClient(mongoClientURI);

                ...
}

$:/path-to-spark-binary/spark-shell --packages org.mongodb.spark:mongo-spark-connector_2.01:2.0.0

import org.apache.spark.{SparkContext, SparkConf}
import com.mongodb.spark.MongoSpark;

sc.stop()

val sc2 = new SparkContext(new SparkConf().setAppName("shell").set("spark.mongodb.input.uri", "mongodb://my_host:my_port/my_database.my_collection"))

MongoSpark.load(sc2)

编辑3：类路径

我在运行时使用以下代码打印了类路径：

ClassLoader cl = ClassLoader.getSystemClassLoader();

    URL[] urls = ((URLClassLoader)cl).getURLs();

    for(URL url: urls){
        System.out.println(url.getFile());
    }

在使用“mongo”作为关键字对结果应用grep后，显示了以下.jar：

$SPARK_HOME/jars/spark-mongodb_2.11-0.12.0.jar
$SPARK_HOME/jars/mongodb-driver-3.4.2.jar
$SPARK_HOME/jars/mongo-hadoop-spark-2.0.2.jar

pom.xml包含以下依赖项：

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.1.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.1.1</version>
    </dependency>
    <dependency>
        <groupId>org.mongodb.spark</groupId>
        <artifactId>mongo-spark-connector_2.11</artifactId>
        <version>2.0.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.11 -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.11</artifactId>
        <version>2.1.1</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>0.10.2.1</version>
    </dependency>
    <dependency>
        <groupId>com.google.code.gson</groupId>
        <artifactId>gson</artifactId>
        <version>2.8.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.cloudera.sparkts/sparkts -->
    <dependency>
        <groupId>com.cloudera.sparkts</groupId>
        <artifactId>sparkts</artifactId>
        <version>0.4.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.drools/drools-compiler -->
    <dependency>
        <groupId>org.drools</groupId>
        <artifactId>drools-compiler</artifactId>
        <version>6.5.0.Final</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.drools/drools-compiler -->
    <dependency>
        <groupId>org.drools</groupId>
        <artifactId>drools-compiler</artifactId>
        <version>6.5.0.Final</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.mongodb/mongo-java-driver -->
    <dependency>
        <groupId>org.mongodb</groupId>
        <artifactId>mongo-java-driver</artifactId>
        <version>3.4.2</version>
    </dependency>


org.apache.spark
spark-core_2.11
2.1.1
org.apache.spark
spark-sql_2.11
2.1.1
org.mongodb.spark
mongo-spark-connector_2.11
2.0.0
org.apache.spark
spark-mllib_2.11
2.1.1
org.apache.kafka
卡夫卡客户
0.10.2.1
com.google.code.gson
格森
2.8.0
com.cloudera.sparkts
斯巴克人
0.4.0
org.drools
drools编译器
6.5.0.1最终版本
org.drools
drools编译器
6.5.0.1最终版本
org.mongodb
mongo java驱动程序
3.4.2

P.S.：无论使用的mongo驱动程序的主要版本是什么，应用程序都会失败，无论它使用的是mongo对象还是MongoClient对象，您的类路径上是否有另一个mongo Java驱动程序版本？您好@Ross，我已经根据您的直觉更新了我的问题@Ross，这似乎是一个类路径问题，我将发布一个完整的答案和解决方案的完整描述，直到明天我还没有发布完整的解决方案，我将在本周这样做，因为我必须在两天内完成这个项目。