Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181

Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/apache-kafka/3.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Apache spark 阅读最新的spark kafka流媒体_Apache Spark_Apache Kafka_Spark Streaming - Fatal编程技术网

Apache spark 阅读最新的spark kafka流媒体

Apache spark 阅读最新的spark kafka流媒体,apache-spark,apache-kafka,spark-streaming,Apache Spark,Apache Kafka,Spark Streaming,我希望使用kafka只阅读spark streaming中的最新消息,但它也会获取过去的数据 如何在KafkaUtil中为spark设置auto.offset.reset JavaPairReceiverInputDStream<String, String> messages = KafkaUtils.createStream(jssc, args[0], args[1], topicMap); JavaPairReceiverInputStream消息=

我希望使用kafka只阅读spark streaming中的最新消息,但它也会获取过去的数据

如何在KafkaUtil中为spark设置auto.offset.reset

JavaPairReceiverInputDStream<String, String> messages =
            KafkaUtils.createStream(jssc, args[0], args[1], topicMap);
JavaPairReceiverInputStream消息=
创建流(jssc,args[0],args[1],topicMap);
如何将conf设置为仅获取当前消息。请举例说明

提前谢谢,还有一个


但是还不够,请帮我解决。提前感谢。

您需要从KafkaUtils对象使用此方法:

 def createStream[K, V, U <: Decoder[_], T <: Decoder[_]](
      jssc: JavaStreamingContext,
      keyTypeClass: Class[K],
      valueTypeClass: Class[V],
      keyDecoderClass: Class[U],
      valueDecoderClass: Class[T],
      kafkaParams: JMap[String, String],
      topics: JMap[String, JInt],
      storageLevel: StorageLevel
    )
def createStream[K,V,U