Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/scala/19.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Java spark2数据帧中的reduceByKeyAndWindow_Java_Scala_Apache Spark_Spark Dataframe_Apache Spark 2.0 - Fatal编程技术网

Java spark2数据帧中的reduceByKeyAndWindow

Java spark2数据帧中的reduceByKeyAndWindow,java,scala,apache-spark,spark-dataframe,apache-spark-2.0,Java,Scala,Apache Spark,Spark Dataframe,Apache Spark 2.0,在Spark 1.6中,使用StreamingContext我可以使用函数ReduceByAndWindow events .mapToPair(x-> new Tuple2<String,MyPojo>(x.getID(),x)) .reduceByKeyAndWindow((a, b) -> a.getQuality() > b.getQuality()

在Spark 1.6中,使用
StreamingContext
我可以使用函数
ReduceByAndWindow

        events
            .mapToPair(x-> new Tuple2<String,MyPojo>(x.getID(),x))
            .reduceByKeyAndWindow((a, b) -> 
                     a.getQuality() > b.getQuality() ? a : b
                , Durations.seconds(properties.getWindowLenght()), 
                  Durations.seconds(properties.getSlidingWindow()))
            .map(y->y._2);
我能够用
groupBy

        events
            .groupBy(functions.window(col("timeStamp"), "10 minutes", "5 minutes"),col("id"))
            .max("quality")
            .join(events, "id");
当我使用groupBy时,我只得到了15列中的两列,因此我尝试使用join将它们恢复,但随后我得到了一个例外:
不支持两个流数据帧/数据集之间的join

我有没有办法在spark 2中重现
ReduceByAndWindow
的逻辑

        events
            .groupBy(functions.window(col("timeStamp"), "10 minutes", "5 minutes"),col("id"))
            .max("quality")
            .join(events, "id");