Java 使用spark streaming连接到Cassandra时出错
我试图使用cassandra创建一个键空间和表,但我得到了一个错误。事实上,我想把spark和cassandra联系起来 我有以下代码:Java 使用spark streaming连接到Cassandra时出错,java,apache-spark,cassandra,apache-kafka,spark-streaming,Java,Apache Spark,Cassandra,Apache Kafka,Spark Streaming,我试图使用cassandra创建一个键空间和表,但我得到了一个错误。事实上,我想把spark和cassandra联系起来 我有以下代码: public static void main(String[] args){ SparkConf sparkConf = new SparkConf().setAppName("org.kakfa.spark.ConsumerData").setMaster("local[4]"); // Substitute 127.0.0.1 with
public static void main(String[] args){
SparkConf sparkConf = new SparkConf().setAppName("org.kakfa.spark.ConsumerData").setMaster("local[4]");
// Substitute 127.0.0.1 with the actual address of your Spark Master (or use "local" to run in local mode
sparkConf.set("spark.cassandra.connection.host", "127.0.0.1");
// Create the context with 2 seconds batch size
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(2000));
Map<String, Integer> topicMap = new HashMap<>();
String[] topics = KafkaProperties.TOPIC.split(",");
for (String topic: topics) {
topicMap.put(topic, KafkaProperties.NUM_THREADS);
}
/* connection to cassandra */
CassandraConnector connector = CassandraConnector.apply(sparkConf);
System.out.println("+++++++++++ cassandra connector created ++++++++++++++++++++++++++++");
/* Receive kafka inputs */
JavaPairReceiverInputDStream<String, String> messages =
KafkaUtils.createStream(jssc, KafkaProperties.ZOOKEEPER, KafkaProperties.GROUP_CONSUMER, topicMap);
System.out.println("+++++++++++++ streaming-kafka connection done +++++++++++++++++++++++++++");
System.out.println(" ----- trying to create tables ------ ");
try (Session session = connector.openSession()) {
session.execute("DROP KEYSPACE IF EXISTS test");
session.execute("CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1}");
session.execute("CREATE TABLE test.users (id TEXT PRIMARY KEY, name TEXT)");
}
System.out.println("---- tables created ----");
起初我以为是主机,但后来我将连接主机更改为“本地”以获取下一个错误,我不知道应该在此处设置什么以避免获取此错误:
Exception in thread "main" java.lang.IllegalArgumentException: Cannot build a cluster without contact points
at com.datastax.driver.core.Cluster.checkNotEmpty(Cluster.java:108)
at com.datastax.driver.core.Cluster.<init>(Cluster.java:100)
at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:169)
at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1031)
at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:179)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:161)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:161)
at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:36)
at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:61)
at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:70)
at org.kakfa.spark.ConsumerData.main(ConsumerData.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
线程“main”java.lang.IllegalArgumentException中的异常:无法构建没有联系人的集群
位于com.datastax.driver.core.Cluster.checkNotEmpty(Cluster.java:108)
位于com.datastax.driver.core.Cluster.(Cluster.java:100)
位于com.datastax.driver.core.Cluster.buildFrom(Cluster.java:169)
位于com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1031)
在com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:179)
在com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply上(CassandraConnector.scala:161)
在com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply上(CassandraConnector.scala:161)
在com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:36)上
位于com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:61)
在com.datasax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:70)
位于org.kakfa.spark.ConsumerData.main(ConsumerData.java:73)
在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)处
在sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)中
在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)中
位于java.lang.reflect.Method.invoke(Method.java:606)
位于com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
在我通过从命令行调用'bin/cassandra-f'启动cassandra之前添加它
谢谢 问题是我有错误的依赖关系
如果您想了解更多信息,可以查看。问题是我的依赖项错误
如果您想了解更多信息,可以查看。是否通过命令行连接到cassandra
cqlsh localhost
当我尝试它时,我得到以下错误连接错误:(“无法连接到任何服务器”,“127.0.0.1”:错误(111,“尝试连接到[('127.0.0.1',9042)]。最后一个错误:连接被拒绝”))
然后需要正确配置cassandra服务器。可能cassandra.yaml
没有包含正确的配置。我现在启动了cassandra,忘记调用bin/cassandra-f,现在我在127.0.0.1:9042连接到测试集群。[cqlsh 5.0.1 | Cassandra 3.0.5 | CQL规范3.4.0 |本机协议v4]使用帮助获取帮助。cqlsh>我再次执行了该程序,但仍然收到相同的错误您是否设法从命令行连接到cassandracqlsh localhost
当我尝试它时,我得到以下错误连接错误:(“无法连接到任何服务器”,“127.0.0.1”:错误(111,“尝试连接到[('127.0.0.1',9042)]。最后一个错误:连接被拒绝”))
然后需要正确配置cassandra服务器。可能cassandra.yaml
没有包含正确的配置。我现在启动了cassandra,忘记调用bin/cassandra-f,现在我在127.0.0.1:9042连接到测试集群。[cqlsh 5.0.1 | Cassandra 3.0.5 | CQL规范3.4.0 |本机协议v4]使用帮助获取帮助。cqlsh>我再次执行了程序,但仍然得到相同的错误
Exception in thread "main" java.lang.IllegalArgumentException: Cannot build a cluster without contact points
at com.datastax.driver.core.Cluster.checkNotEmpty(Cluster.java:108)
at com.datastax.driver.core.Cluster.<init>(Cluster.java:100)
at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:169)
at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1031)
at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:179)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:161)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:161)
at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:36)
at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:61)
at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:70)
at org.kakfa.spark.ConsumerData.main(ConsumerData.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)