Warning: file_get_contents(/data/phpspider/zhask/data//catemap/9/java/336.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Java 启动spark上下文时出错-未注册任何输出操作,因此无法执行任何操作_Java_Apache Spark_Spark Streaming_Kafka Consumer Api_Bigdata - Fatal编程技术网

Java 启动spark上下文时出错-未注册任何输出操作,因此无法执行任何操作

Java 启动spark上下文时出错-未注册任何输出操作,因此无法执行任何操作,java,apache-spark,spark-streaming,kafka-consumer-api,bigdata,Java,Apache Spark,Spark Streaming,Kafka Consumer Api,Bigdata,我试图打开一个火花流,听一个卡夫卡的主题,并做一些基本的逻辑。我遵循了教程,并打算进行简单的字数计算。当执行下面的代码时,我不断得到相同的错误 16/03/29 11:21:40 ERROR StreamingContext: Error starting the context, marking it as stopped java.lang.IllegalArgumentException: requirement failed: No output operations registere

我试图打开一个火花流,听一个卡夫卡的主题,并做一些基本的逻辑。我遵循了教程,并打算进行简单的字数计算。当执行下面的代码时,我不断得到相同的错误

16/03/29 11:21:40 ERROR StreamingContext: Error starting the context, marking it as stopped
java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute
    at scala.Predef$.require(Predef.scala:233)
但是,正如您在下面的代码片段中所看到的那样,每次操作之后,我都在使用
print()
函数,因此应该注册这些函数。我还有
start()
awaitTermination()
函数在末尾,所以一切都应该井然有序。我使用的是spark版本1.4.1(流媒体和kafka流媒体依赖项)

//创建流上下文
SparkConf conf=new SparkConf().setMaster(this.getApplicationMode()).setAppName(this.getApplicationName());
JavaStreamingContext ssc=newJavaStreamingContext(conf,new Duration(this.getBatchDuration());
//仅当我们指定时添加检查点。。。
if(this.getCheckpointPath()!=null){
ssc.checkpoint(this.getCheckpointPath());
}
//准备配置,例如要收听的主题,以及kafka代理列表
HashSet topicsSet=新的HashSet(Arrays.asList(this.getKafkaTopicName().split(“,”));
HashMap kafkaParams=新HashMap();
kafkaParams.put(“metadata.broker.list”,this.getBrokerList());
//创建带有代理和主题的直接卡夫卡流
JavaPairInputStream消息=KafkaUtils.createDirectStream(
ssc,
String.class,
String.class,
kafka.serializer.StringDecoder.class,
kafka.serializer.StringDecoder.class,
卡夫卡帕拉姆斯,
主题集
);
messages.print();
JavadStreamLines=messages.map(新函数(){
@凌驾
公共字符串调用(Tuple2 Tuple2){
返回tuple2._2();
}
});
line.print();
JavaDStream words=lines.flatMap(新的flatMap函数(){
@凌驾
公共Iterable调用(字符串s)引发异常{
TreeSet TreeSet=新树集();
树集。添加(s);
Iterable it=树集;
归还它;
}
});
words.print();
JavaPairDStream wordCounts=words.mapToPair(
新PairFunction(){
@凌驾
公共元组2调用(字符串s){
返回新的Tuple2(s,1);
}
}).reduceByKey(新功能2(){
@凌驾
公共整数调用(整数i1、整数i2){
返回i1+i2;
}
});
wordCounts.print();
//扳机!
ssc.start();
//等待服务停止。。。
ssc.终止();

关于我可能遗漏的内容,有什么建议吗?

您找到解决方案了吗?
        // Create streaming context
    SparkConf conf = new SparkConf().setMaster(this.getApplicationMode()).setAppName(this.getApplicationName()); 
    JavaStreamingContext ssc = new JavaStreamingContext(conf, new Duration(this.getBatchDuration()));

    // Add checkpoint only if we specified...
    if(this.getCheckpointPath() != null) {
        ssc.checkpoint(this.getCheckpointPath());
    }

    // Prepare configurations, such as topic to listen to, and also the kafka broker list
    HashSet<String> topicsSet = new HashSet<>(Arrays.asList(this.getKafkaTopicName().split(",")));
    HashMap<String, String> kafkaParams = new HashMap<>();
    kafkaParams.put("metadata.broker.list", this.getBrokerList());

    // Create direct kafka stream with brokers and topics
    JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream(
            ssc,
            String.class,
            String.class,
            kafka.serializer.StringDecoder.class,
            kafka.serializer.StringDecoder.class,
            kafkaParams,
            topicsSet
    );

    messages.print();

    JavaDStream<String> lines = messages.map(new Function<Tuple2<String, String>, String>() {
        @Override
        public String call(Tuple2<String, String> tuple2) {
            return tuple2._2();
        }
    });

    lines.print();

    JavaDStream<String> words = lines.flatMap(new FlatMapFunction<String, String>() {

        @Override
        public Iterable<String> call(String s) throws Exception {
            TreeSet treeSet = new TreeSet<String>();
            treeSet.add(s);

            Iterable<String> it = treeSet;

            return it;
        }
    });

    words.print();

    JavaPairDStream<String, Integer> wordCounts = words.mapToPair(
            new PairFunction<String, String, Integer>() {
                @Override
                public Tuple2<String, Integer> call(String s) {
                    return new Tuple2<>(s, 1);
                }
            }).reduceByKey(new Function2<Integer, Integer, Integer>() {
        @Override
        public Integer call(Integer i1, Integer i2) {
            return i1 + i2;
        }
    });

    wordCounts.print();

    // Trigger!
    ssc.start();

    // Await stopping of the service...
    ssc.awaitTermination();