Apache spark 卡夫卡充满了火花。依赖性问题
我尝试将Spark Streaming 2.0.0与Kafka 0.10结合使用。我正在使用集成测试,但启动服务器时遇到一些问题。当我尝试使用Spark 2.2.0时,它工作正常Apache spark 卡夫卡充满了火花。依赖性问题,apache-spark,apache-kafka,Apache Spark,Apache Kafka,我尝试将Spark Streaming 2.0.0与Kafka 0.10结合使用。我正在使用集成测试,但启动服务器时遇到一些问题。当我尝试使用Spark 2.2.0时,它工作正常 <dependency> <groupId>net.manub</groupId> <artifactId>scalatest-embedded-kafka_2.11</artifactId> <ve
<dependency>
<groupId>net.manub</groupId>
<artifactId>scalatest-embedded-kafka_2.11</artifactId>
<version>${embedded-kafka.version}</version> -->I tried many versions.
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<version>2.0.2</version>
</dependency>
An exception or error caused a run to abort: kafka.server.KafkaServer$.$lessinit$greater$default$2()Lorg/apache/kafka/common/utils/Time;
java.lang.NoSuchMethodError: kafka.server.KafkaServer$.$lessinit$greater$default$2()Lorg/apache/kafka/common/utils/Time;
at net.manub.embeddedkafka.EmbeddedKafkaSupport$class.startKafka(EmbeddedKafka.scala:467)
at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:38)
at net.manub.embeddedkafka.EmbeddedKafka$.start(EmbeddedKafka.scala:55)
at iris.orange.ScalaTest$$anonfun$1.apply$mcV$sp(ScalaTest.scala:10)
net.manub
scalatest-embedded-kafka_2.11
${embedded kafka.version}-->我尝试了许多版本。
测验
org.apache.spark
spark-streaming-kafka-0-10_2.11
2.0.2
异常或错误导致运行中止:kafka.server.KafkaServer$.$lessinit$greater$default$2()Lorg/apache/kafka/common/utils/Time;
java.lang.NoSuchMethodError:kafka.server.KafkaServer$.$lessinit$greater$default$2()Lorg/apache/kafka/common/utils/Time;
位于net.manub.embeddedkafka.EmbeddedKafkaSupport$class.startKafka(embeddedkafka.scala:467)
位于net.manub.embeddedkafka.embeddedkafka$.startKafka(embeddedkafka.scala:38)
位于net.manub.embeddedkafka.embeddedkafka$.start(embeddedkafka.scala:55)
在iris.orange.ScalaTest$$anonfun$1.apply$mcV$sp(ScalaTest.scala:10)
这似乎是一个关于依赖性的问题,但我没有开始工作。我选择了一个使用相同卡夫卡版本的嵌入式卡夫卡 您需要使用正确版本的
spark streaming kafka
org.apache.spark
spark-streaming-kafka-0-10_2.10
2.0.0
您需要获得spark版本的特定版本
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.10</artifactId>
<version>2.0.0</version>
</dependency>