Maven Intellij无法识别sparksession sparkcontext sqlcontext-Spark版本2.4.4
我正在用spark和scala在Intellij中创建Maven项目。但Intellij未能识别sparksession、sparkcontext和sqlcontext。随附图片 错误:无法解析符号getsqlcontext 错误:无法解析符号getSparkContaxt 我对POM的理解是项目对象模型,或者POM是Maven中的基本工作单元。它是一个XML文件,包含有关项目的信息和Maven用于构建项目的配置详细信息 我是否需要更改POM属性或依赖项中的某些内容 我的POM文件看起来像这样Maven Intellij无法识别sparksession sparkcontext sqlcontext-Spark版本2.4.4,maven,apache-spark,intellij-idea,spark-cassandra-connector,Maven,Apache Spark,Intellij Idea,Spark Cassandra Connector,我正在用spark和scala在Intellij中创建Maven项目。但Intellij未能识别sparksession、sparkcontext和sqlcontext。随附图片 错误:无法解析符号getsqlcontext 错误:无法解析符号getSparkContaxt 我对POM的理解是项目对象模型,或者POM是Maven中的基本工作单元。它是一个XML文件,包含有关项目的信息和Maven用于构建项目的配置详细信息 我是否需要更改POM属性或依赖项中的某些内容 我的POM文件看起来像这样
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.IPM</groupId>
<artifactId>lambda</artifactId>
<packaging>pom</packaging>
<version>1.0-SNAPSHOT</version>
<modules>
<module>spark-lambda</module>
</modules>
<properties>
<spark.version>2.4.4</spark.version>
<chill.version>0.7.2</chill.version>
<algebird.version>0.11.0</algebird.version>
<kafka.clients.version>0.8.2.1</kafka.clients.version>
<avro.version>1.7.7</avro.version>
<spark-cassandra-connector.version>1.6.1</spark-cassandra-connector.version>
<cassandra-driver-core.version>3.0.1</cassandra-driver-core.version>
<nscala-time.version>2.6.0</nscala-time.version>
</properties>
<dependencies>
<dependency>
<groupId>com.typesafe</groupId>
<artifactId>config</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.11</artifactId>
<version>${spark-cassandra-connector.version}</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>${cassandra-driver-core.version}</version>
</dependency>
<dependency>
<groupId>com.github.nscala-time</groupId>
<artifactId>nscala-time_2.11</artifactId>
<version>${nscala-time.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>${kafka.clients.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>com.twitter</groupId>
<artifactId>chill_2.11</artifactId>
<version>${chill.version}</version>
</dependency>
<dependency>
<groupId>com.twitter</groupId>
<artifactId>chill-avro_2.11</artifactId>
<version>${chill.version}</version>
</dependency>
<dependency>
<groupId>com.twitter</groupId>
<artifactId>algebird-core_2.11</artifactId>
<version>${algebird.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
<!-- provided -->
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>${kafka.clients.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
<!-- provided -->
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
<!-- provided -->
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
<!-- provided -->
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
<!-- provided -->
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.8.1</version>
</dependency>
<dependency>
<groupId>org.joda</groupId>
<artifactId>joda-convert</artifactId>
<version>1.7</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.1.6</version>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<artifactSet>
<includes>
<include>*:*</include>
</includes>
</artifactSet>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
4.0.0
com.IPM
Spark中没有此类函数,您需要正确创建上下文,如下所示(如下所示):
def main(args:Array[String]):单位={
val sc=新的SparkContext()
val spark=SparkSession.builder().config(sc.getConf.getOrCreate())
导入spark.implicits_
...
}
第二,您的依赖项不正确-您正在使用Spark Cassandra Connector 1.6.1,它是为Spark 1.6设计的,而您正在尝试使用Spark 2.4.4
您需要使用Spark Cassandra Connector 2.5.0(最新版本)-它与Spark 2.4兼容,并且还具有
您还需要删除此依赖项:
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>${cassandra-driver-core.version}</version>
</dependency>
com.datasax.cassandra
使用Kafka+Spark+Cassandra的方法。Spark中没有此类函数,您需要正确创建上下文,如下所示(如下所示):
def main(args:Array[String]):单位={
val sc=新的SparkContext()
val spark=SparkSession.builder().config(sc.getConf.getOrCreate())
导入spark.implicits_
...
}
第二,您的依赖项不正确-您正在使用Spark Cassandra Connector 1.6.1,它是为Spark 1.6设计的,而您正在尝试使用Spark 2.4.4
您需要使用Spark Cassandra Connector 2.5.0(最新版本)-它与Spark 2.4兼容,并且还具有
您还需要删除此依赖项:
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>${cassandra-driver-core.version}</version>
</dependency>
com.datasax.cassandra
使用Kafka+Spark+Cassandra的可能性。你有太多的依赖项,imho。从小处做起。抓取东西-它的工作项目。。。只需克隆repo,从中复制Pom.xml,稍微调整版本以使用SCC 2.5.0您有太多的依赖项,imho。从小处做起。抓取东西-它的工作项目。。。只需克隆repo,从中复制Pom.xml,稍微调整版本以使用SCC2.5.0