Warning: file_get_contents(/data/phpspider/zhask/data//catemap/9/java/346.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
火花检查点不存在';不记得状态(Java HDFS)_Java_Apache Spark_Hdfs_Spark Streaming_Hadoop2 - Fatal编程技术网

火花检查点不存在';不记得状态(Java HDFS)

火花检查点不存在';不记得状态(Java HDFS),java,apache-spark,hdfs,spark-streaming,hadoop2,Java,Apache Spark,Hdfs,Spark Streaming,Hadoop2,已经看了 但是没有帮助。 虽然我使用的是spark streaming 2.11 v 2.0.1,但也查看了JavaStreamingContextFactory,但找不到 我的代码工作正常,但当我重新启动它。。。它不会记得最后一个检查点 Function0<JavaStreamingContext> scFunction = new Function0<JavaStreamingContext>() { @Override public

已经看了 但是没有帮助。 虽然我使用的是spark streaming 2.11 v 2.0.1,但也查看了JavaStreamingContextFactory,但找不到

我的代码工作正常,但当我重新启动它。。。它不会记得最后一个检查点

Function0<JavaStreamingContext> scFunction = new Function0<JavaStreamingContext>() {
        @Override
        public JavaStreamingContext call() throws Exception {
            //Spark Streaming needs to checkpoint enough information to a fault- tolerant storage system such
            JavaStreamingContext ssc = new JavaStreamingContext(conf, Durations.milliseconds(SPARK_DURATION));
          //checkpointDir = "hdfs://user:pw@192.168.1.50:54310/spark/checkpoint";
            ssc.sparkContext().setCheckpointDir(checkpointDir);
            StorageLevel.MEMORY_AND_DISK();
            return ssc;
        }
    };

    JavaStreamingContext ssc = JavaStreamingContext.getOrCreate(checkpointDir, scFunction);
Function0 scFunction=newfunction0(){
@凌驾
公共JavaStreamingContext调用()引发异常{
//Spark Streaming需要为容错存储系统检查足够的信息,例如
JavaStreamingContext ssc=新的JavaStreamingContext(conf,Durations.millizes(SPARK_DURATION));
//检查点目录=”hdfs://user:pw@192.168.1.50:54310/火花/检查点”;
ssc.sparkContext().setCheckpointDir(checkpointDir);
StorageLevel.MEMORY_和_DISK();
返回ssc;
}
};
JavaStreamingContext ssc=JavaStreamingContext.getOrCreate(checkpointDir,scFunction);
目前数据正在从卡夫卡流式传输,我正在执行一些转换和操作

JavaPairDStream<Integer, Long> responseCodeCountDStream = logObject.transformToPair
            (MainApplication::responseCodeCount);
    JavaPairDStream<Integer, Long> cumulativeResponseCodeCountDStream = responseCodeCountDStream.updateStateByKey
            (COMPUTE_RUNNING_SUM);
    cumulativeResponseCodeCountDStream.foreachRDD(rdd -> {
        rdd.checkpoint();
        LOG.warn("Response code counts: " + rdd.take(100));
    });
JavaPairDStream responseCodeCountDStream=logObject.transformToPair
(主应用程序::ResponseDeCount);
JavaPairDStream cumulativeResponseCodeCountDStream=responseCodeCountDStream.updateStateByKey
(计算运行和);
累积响应代码计数流foreachRDD(rdd->{
检查点();
LOG.warn(“响应代码计数:+rdd.take(100));
});
如果我遗漏了什么,有人能给我指出正确的方向吗

此外,我可以看到检查点被保存在hdfs中。但是为什么它不从中读出呢