Java 启动spark上下文时出错-未注册任何输出操作,因此无法执行任何操作
我试图打开一个火花流,听一个卡夫卡的主题,并做一些基本的逻辑。我遵循了教程,并打算进行简单的字数计算。当执行下面的代码时,我不断得到相同的错误Java 启动spark上下文时出错-未注册任何输出操作,因此无法执行任何操作,java,apache-spark,spark-streaming,kafka-consumer-api,bigdata,Java,Apache Spark,Spark Streaming,Kafka Consumer Api,Bigdata,我试图打开一个火花流,听一个卡夫卡的主题,并做一些基本的逻辑。我遵循了教程,并打算进行简单的字数计算。当执行下面的代码时,我不断得到相同的错误 16/03/29 11:21:40 ERROR StreamingContext: Error starting the context, marking it as stopped java.lang.IllegalArgumentException: requirement failed: No output operations registere
16/03/29 11:21:40 ERROR StreamingContext: Error starting the context, marking it as stopped
java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute
at scala.Predef$.require(Predef.scala:233)
但是,正如您在下面的代码片段中所看到的那样,每次操作之后,我都在使用print()
函数,因此应该注册这些函数。我还有start()
和awaitTermination()
函数在末尾,所以一切都应该井然有序。我使用的是spark版本1.4.1(流媒体和kafka流媒体依赖项)
//创建流上下文
SparkConf conf=new SparkConf().setMaster(this.getApplicationMode()).setAppName(this.getApplicationName());
JavaStreamingContext ssc=newJavaStreamingContext(conf,new Duration(this.getBatchDuration());
//仅当我们指定时添加检查点。。。
if(this.getCheckpointPath()!=null){
ssc.checkpoint(this.getCheckpointPath());
}
//准备配置,例如要收听的主题,以及kafka代理列表
HashSet topicsSet=新的HashSet(Arrays.asList(this.getKafkaTopicName().split(“,”));
HashMap kafkaParams=新HashMap();
kafkaParams.put(“metadata.broker.list”,this.getBrokerList());
//创建带有代理和主题的直接卡夫卡流
JavaPairInputStream消息=KafkaUtils.createDirectStream(
ssc,
String.class,
String.class,
kafka.serializer.StringDecoder.class,
kafka.serializer.StringDecoder.class,
卡夫卡帕拉姆斯,
主题集
);
messages.print();
JavadStreamLines=messages.map(新函数(){
@凌驾
公共字符串调用(Tuple2 Tuple2){
返回tuple2._2();
}
});
line.print();
JavaDStream words=lines.flatMap(新的flatMap函数(){
@凌驾
公共Iterable调用(字符串s)引发异常{
TreeSet TreeSet=新树集();
树集。添加(s);
Iterable it=树集;
归还它;
}
});
words.print();
JavaPairDStream wordCounts=words.mapToPair(
新PairFunction(){
@凌驾
公共元组2调用(字符串s){
返回新的Tuple2(s,1);
}
}).reduceByKey(新功能2(){
@凌驾
公共整数调用(整数i1、整数i2){
返回i1+i2;
}
});
wordCounts.print();
//扳机!
ssc.start();
//等待服务停止。。。
ssc.终止();
关于我可能遗漏的内容,有什么建议吗?您找到解决方案了吗?
// Create streaming context
SparkConf conf = new SparkConf().setMaster(this.getApplicationMode()).setAppName(this.getApplicationName());
JavaStreamingContext ssc = new JavaStreamingContext(conf, new Duration(this.getBatchDuration()));
// Add checkpoint only if we specified...
if(this.getCheckpointPath() != null) {
ssc.checkpoint(this.getCheckpointPath());
}
// Prepare configurations, such as topic to listen to, and also the kafka broker list
HashSet<String> topicsSet = new HashSet<>(Arrays.asList(this.getKafkaTopicName().split(",")));
HashMap<String, String> kafkaParams = new HashMap<>();
kafkaParams.put("metadata.broker.list", this.getBrokerList());
// Create direct kafka stream with brokers and topics
JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream(
ssc,
String.class,
String.class,
kafka.serializer.StringDecoder.class,
kafka.serializer.StringDecoder.class,
kafkaParams,
topicsSet
);
messages.print();
JavaDStream<String> lines = messages.map(new Function<Tuple2<String, String>, String>() {
@Override
public String call(Tuple2<String, String> tuple2) {
return tuple2._2();
}
});
lines.print();
JavaDStream<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
@Override
public Iterable<String> call(String s) throws Exception {
TreeSet treeSet = new TreeSet<String>();
treeSet.add(s);
Iterable<String> it = treeSet;
return it;
}
});
words.print();
JavaPairDStream<String, Integer> wordCounts = words.mapToPair(
new PairFunction<String, String, Integer>() {
@Override
public Tuple2<String, Integer> call(String s) {
return new Tuple2<>(s, 1);
}
}).reduceByKey(new Function2<Integer, Integer, Integer>() {
@Override
public Integer call(Integer i1, Integer i2) {
return i1 + i2;
}
});
wordCounts.print();
// Trigger!
ssc.start();
// Await stopping of the service...
ssc.awaitTermination();