未能实例化org.apache.spark.api.java.JavaSparkContext

我正在与Springboot和Cassandarab进行ApacheSpark集成。我提供了以下配置文件来配置spark和cassandra db

    @Configuration
    @PropertySource("classpath:application.properties")
    public class SparkConfig {


 private static Logger log = LoggerFactory.getLogger(DashboardRestServicesApplication.class.getName());

 @Value("${spark.master}")
 private String sparkMaster;

 @Value("${spring.data.cassandra.keyspace-name}")
 private String cassandraKeyspace;

 @Value("${cassandra.table}")
 private String cassandraTable;

 @Value("${spring.data.cassandra.contact-points}")
 private String cassandraHost;

 @Value("${spring.data.cassandra.port}")
 private String cassandraPort;

 @Value("${spring.data.cassandra.username}")
 private String username;

 @Value("${spring.data.cassandra.password}")
 private String password;


 @Bean
 public SparkConf sparkConf() {
     SparkConf conf = new SparkConf(true)
             .set("spark.cassandra.connection.host",cassandraHost)
             .set("spark.cassandra.connection.port", cassandraPort)
             .set("spark.cassandra.auth.username", username)
             .set("spark.cassandra.auth.password", password)
             .set("spark.submit.deployMode", "client")
             .setMaster(sparkMaster)
             .setAppName("DashboardSparkService");
     return conf;
 }

 @Bean
 public JavaSparkContext javaSparkContext() {
     log.info("Connecting to spark with master Url: {}, and cassandra host: {}",
                sparkMaster, cassandraHost);

     JavaSparkContext javaSparkContext = new JavaSparkContext(sparkConf());

     log.debug("spark context created");

     return javaSparkContext;
 }


 @Bean
 public SparkSession sparkSession() {
     return SparkSession
             .builder()
             .sparkContext(javaSparkContext().sc())
             .appName("DashboardSparkService")
             .getOrCreate();
 }

 }

这是我的服务：

  @Autowired
private JavaSparkContext javaSparkContext;

@Autowired
private TaskSummarySparkRepo taskSummarySparkRepo;

public int getAllOrders() {
    //JavaSparkContext javaSparkContext = sparkConfig.javaSparkContext();
    return taskSummarySparkRepo.getAllOrders(javaSparkContext);
}

这是我的回购协议：

    @Service                                     
    public class TaskSummarySparkRepo {

    @Value("${spring.data.cassandra.keyspace-name}")
    private String CassandraKeyspace;

    @Value("${cassandra.table}")
    private String CassandraTable;


    public int getAllOrders(JavaSparkContext javaSparkContext) {
          JavaRDD<TaskSummary> rdd = javaFunctions(javaSparkContext)
                    .cassandraTable(CassandraKeyspace, CassandraTable, 
                     mapRowTo(TaskSummary.class));

          return (int) rdd.count();
    }
}

当我尝试从sts将其作为spring boot应用程序运行时，会出现以下错误：

    org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'taskSummarySparkService': Unsatisfied dependency expressed through field 'javaSparkContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'javaSparkContext' defined in class path resource [com/spectrum/dashboard/config/SparkConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.spark.api.java.JavaSparkContext]: Factory method 'javaSparkContext' threw exception; nested exception is java.lang.NoClassDefFoundError: org/apache/spark/network/util/ByteUnit

    Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'javaSparkContext' defined in class path resource [com/spectrum/dashboard/config/SparkConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.spark.api.java.JavaSparkContext]: Factory method 'javaSparkContext' threw exception; nested exception is java.lang.NoClassDefFoundError: org/spark_project/guava/collect/MapMaker        

编辑： 以下是我在项目中使用的pom文件依赖项：

    <dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>

    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-data-cassandra</artifactId>
    </dependency>

    <!-- https://mvnrepository.com/artifact/com.datastax.cassandra/cassandra-driver-core -->
    <dependency>
        <groupId>com.datastax.cassandra</groupId>
        <artifactId>cassandra-driver-core</artifactId>
        <version>3.5.0</version>
    </dependency>



    <!-- https://mvnrepository.com/artifact/com.datastax.cassandra/cassandra-driver-mapping -->
    <!--   <dependency>
        <groupId>com.datastax.cassandra</groupId>
        <artifactId>cassandra-driver-mapping</artifactId>
        <version>3.5.0</version>
    </dependency>
    -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>


    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.2.1</version>
    </dependency>




    <!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector -->
    <dependency>
        <groupId>com.datastax.spark</groupId>
        <artifactId>spark-cassandra-connector_2.11</artifactId>
        <version>2.0.8</version>
    </dependency>


    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.2.1</version>
    </dependency>

</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
</build>

org.springframework.boot
弹簧靴起动器
org.springframework.boot
弹簧起动试验
测试
org.springframework.data
弹簧数据卡桑德拉
com.datasax.cassandra
卡桑德拉驱动核心
3.5.0
org.springframework.boot
SpringBootStarterWeb
org.apache.spark
spark-core_2.11
2.2.1
com.datasax.spark
spark-cassandra-connector_2.11
2.0.8
org.apache.spark
spark-sql_2.11
2.2.1
org.springframework.boot
springbootmaven插件

配置有问题吗？还是因为依赖项注入的问题？

---

感谢您的帮助

当我删除现有的maven存储库，然后执行maven安装和机器重启时，问题得到了解决

这是一个类路径问题。您正在使用什么构建工具？专家格拉德尔？发布依赖项部分。在类路径上有多个Spark库吗？我使用maven作为构建工具。我在pom中包含了spark sql、spark cassandra connector和spark core依赖项。Spring-boot无法初始化SparkConfig文件中存在的javaSparkContext bean。我在javaSparkContext对象中提供了连接cassandra db和连接spark master的配置，看起来不错。错误的来源是什么？

java.lang.NoClassDefFoundError

几乎总是指向类路径问题。您最近的一个引用了番石榴，您可能有一个spark/Guava组合，其中的依赖项彼此不兼容。是的，这是番石榴的依赖项问题。日志显示，在org/spark_project/guava/collect/MapMaker中找不到该类，但它存在于maven依赖项中。Mapmaker类是spark网络公共jar文件的一部分，该文件附带spark核心依赖项。我的问题是，如果一个文件存在于maven依赖项中，那么为什么spring boot会显示在运行时找不到的类呢。