Java Spark 1.5和datastax-ddc-3.2.1 Cassandra依赖性罐？_Java_Guava_Spark Cassandra Connector_Cassandra 3.0

Java Spark 1.5和datastax-ddc-3.2.1 Cassandra依赖性罐？

java

Java Spark 1.5和datastax-ddc-3.2.1 Cassandra依赖性罐？,java,guava,spark-cassandra-connector,cassandra-3.0,Java,Guava,Spark Cassandra Connector,Cassandra 3.0,我使用的是Spark 1.5和Cassandra 3.2.1。任何人都可以指定构建路径中需要存在哪些JAR来连接、查询和插入数据到Cassandra 现在我正在使用下面的罐子 spark-cassandra-connector_2.10-1.5.0-M3.jar apache-cassandra-clientutil-3.2.1.jar cassandra-driver-core-3.0.0-beta1-bb1bce4-SNAPSHOT-shaded.jar spark-assembly-1.5

我使用的是Spark 1.5和Cassandra 3.2.1。任何人都可以指定构建路径中需要存在哪些JAR来连接、查询和插入数据到Cassandra

现在我正在使用下面的罐子 spark-cassandra-connector_2.10-1.5.0-M3.jar apache-cassandra-clientutil-3.2.1.jar cassandra-driver-core-3.0.0-beta1-bb1bce4-SNAPSHOT-shaded.jar spark-assembly-1.5.1-hadoop2.0.0-mr1-cdh4.2.0.jar 番石榴-18.0.jar netty-all-4.0.23.Final.jar

有了以上的罐子，我可以连接到卡桑德拉。我可以截断表，删除表。但我无法插入任何数据，即使是简单的插入查询

代码如下：

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;

import com.datastax.driver.core.Session;
import com.datastax.spark.connector.cql.CassandraConnector;

public class Test {

public static void main(String[] args) {


    JavaSparkContext ctx = new JavaSparkContext(new SparkConf().setMaster("spark://blr-lt-203:7077").set("spark.cassandra.connection.host", "blr-lt-203").setAppName("testinsert").set("spark.serializer" ,"org.apache.spark.serializer.KryoSerializer").set("spark.kryoserializer.buffer.max" , "1024mb"));

    CassandraConnector connector = CassandraConnector.apply(ctx.getConf());

    Session session = connector.openSession();

    session.execute("insert into test.table1 (name) values ('abcd')") ;
    session.close();
    ctx.stop();

}

}

以下是日志：

16/03/28 21:24:52 INFO BlockManagerMaster: Trying to register BlockManager
16/03/28 21:24:52 INFO BlockManagerMasterEndpoint: Registering   block    manager localhost:50238 with 944.7 MB RAM,BlockManagerId(driver, localhost, 50238)
16/03/28 21:24:52 INFO BlockManagerMaster: Registered BlockManager
16/03/28 21:24:53 INFO NettyUtil: Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
16/03/28 21:24:53 INFO Cluster: New Cassandra host localhost/127.0.0.1:9042 added
16/03/28 21:24:53 INFO CassandraConnector: Connected to Cassandra cluster: Test Cluster

它只是在这里停了一段时间，然后超时，下面是一个例外：

Exception in thread "main" com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)

我做错了什么

请告诉我所需的JAR是什么，或者是否存在版本兼容性问题

spark（1.5）和cassandra（？）最稳定的版本是什么

提前感谢

这个问题是由于谷歌的番石榴库之间的冲突造成的

解决方案是对spark cassandra连接器依赖项中的番石榴库进行着色。您可以使用maven shade插件来实现这一点。这是我的pom.xml，它可以遮住guava库

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0  
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.pc.test</groupId>
<artifactId>casparktest</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>

 <name>casparktest</name>
<url>http://maven.apache.org</url>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<dependencies>
<dependency>
 <groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <version>3.8.1</version>
  <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.datastax.spark</groupId>
    <artifactId>spark-cassandra-connector_2.10</artifactId>
    <version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.0.0-beta1</version>
</dependency>

</dependencies>
<build>

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.3</version>
        <executions>
            <execution>
                <phase>package</phase>
                <goals>
                    <goal>shade</goal>
                </goals>
                <configuration>
                 <filters>
    <filter>
        <artifact>*:*</artifact>
        <excludes>
            <exclude>META-INF/*.SF</exclude>
            <exclude>META-INF/*.DSA</exclude>
            <exclude>META-INF/*.RSA</exclude>
        </excludes>
    </filter>
</filters>
                    <relocations>
                        <relocation>
                            <pattern>com.google</pattern>
                            <shadedPattern>com.pointcross.shaded.google</shadedPattern>
                        </relocation>

                    </relocations>
                    <minimizeJar>false</minimizeJar>
                    <shadedArtifactAttached>true</shadedArtifactAttached>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>
</build>


4.0.0
com.pc.test
卡斯帕克试验
0.0.1-快照
罐子
卡斯帕克试验
http://maven.apache.org
UTF-8
org.apache.spark
spark-core_2.10
1.5.0
朱尼特
朱尼特
3.8.1
测试
com.datasax.spark
spark-cassandra-connector_2.10
1.5.0
com.datasax.cassandra
卡桑德拉驱动核心
3.0.0-beta1
org.apache.maven.plugins
maven阴影插件
2.3
包裹
阴凉处
*:*
META-INF/*.SF
META-INF/*.DSA
META-INF/*.RSA
谷歌
com.pointcross.shade.google
假的
真的

之后，您将执行maven构建，该构建将生成一个jar，其中包含pom.xml中提到的所有依赖项，并对guava库进行着色，您可以使用这些库提交spark作业。

由于google的guava库之间存在冲突，因此出现了问题

解决方案是对spark cassandra连接器依赖项中的番石榴库进行着色。您可以使用maven shade插件来实现这一点。这是我的pom.xml，它可以遮住guava库

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0  
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.pc.test</groupId>
<artifactId>casparktest</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>

 <name>casparktest</name>
<url>http://maven.apache.org</url>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<dependencies>
<dependency>
 <groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <version>3.8.1</version>
  <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.datastax.spark</groupId>
    <artifactId>spark-cassandra-connector_2.10</artifactId>
    <version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.0.0-beta1</version>
</dependency>

</dependencies>
<build>

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.3</version>
        <executions>
            <execution>
                <phase>package</phase>
                <goals>
                    <goal>shade</goal>
                </goals>
                <configuration>
                 <filters>
    <filter>
        <artifact>*:*</artifact>
        <excludes>
            <exclude>META-INF/*.SF</exclude>
            <exclude>META-INF/*.DSA</exclude>
            <exclude>META-INF/*.RSA</exclude>
        </excludes>
    </filter>
</filters>
                    <relocations>
                        <relocation>
                            <pattern>com.google</pattern>
                            <shadedPattern>com.pointcross.shaded.google</shadedPattern>
                        </relocation>

                    </relocations>
                    <minimizeJar>false</minimizeJar>
                    <shadedArtifactAttached>true</shadedArtifactAttached>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>
</build>


4.0.0
com.pc.test
卡斯帕克试验
0.0.1-快照
罐子
卡斯帕克试验
http://maven.apache.org
UTF-8
org.apache.spark
spark-core_2.10
1.5.0
朱尼特
朱尼特
3.8.1
测试
com.datasax.spark
spark-cassandra-connector_2.10
1.5.0
com.datasax.cassandra
卡桑德拉驱动核心
3.0.0-beta1
org.apache.maven.plugins
maven阴影插件
2.3
包裹
阴凉处
*:*
META-INF/*.SF
META-INF/*.DSA
META-INF/*.RSA
谷歌
com.pointcross.shade.google
假的
真的

之后，您将进行maven构建，生成一个jar，其中包含pom.xml中提到的所有依赖项，并对您可以提交spark作业的guava库进行着色处理。

我也面临同样的问题，您的问题解决了吗？我也面临同样的问题，您的问题解决了吗？？