Scala 火花流窗口操作
下面是获取30秒窗口大小和10秒幻灯片大小上的字数的简单代码Scala 火花流窗口操作,scala,distributed,apache-spark,spark-streaming,Scala,Distributed,Apache Spark,Spark Streaming,下面是获取30秒窗口大小和10秒幻灯片大小上的字数的简单代码 import org.apache.spark.SparkConf import org.apache.spark.streaming._ import org.apache.spark.streaming.StreamingContext._ import org.apache.spark.api.java.function._ import org.apache.spark.streaming.api._ import org.a
import org.apache.spark.SparkConf
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.api.java.function._
import org.apache.spark.streaming.api._
import org.apache.spark.storage.StorageLevel
val ssc = new StreamingContext(sc, Seconds(5))
// read from text file
val lines0 = ssc.textFileStream("test")
val words0 = lines0.flatMap(_.split(" "))
// read from socket
val lines1 = ssc.socketTextStream("localhost", 9999, StorageLevel.MEMORY_AND_DISK_SER)
val words1 = lines1.flatMap(_.split(" "))
val words = words0.union(words1)
val wordCounts = words.map((_, 1)).reduceByKeyAndWindow(_ + _, Seconds(30), Seconds(10))
wordCounts.print()
ssc.checkpoint(".")
ssc.start()
ssc.awaitTermination()
但是,我从这一行中得到了错误:
val wordCounts = words.map((_, 1)).reduceByKeyAndWindow(_ + _, Seconds(30), Seconds(10))
。特别是,从\uu+\uu
。错误是
51: error: missing parameter type for expanded function ((x$2, x$3) => x$2.$plus(x$3))
有人能告诉我问题出在哪里吗?谢谢 这非常容易修复,只需明确说明类型。
val-wordCounts=words.map((u,1)).reduceByKeyAndWindow((a:Int,b:Int)=>a+b,秒(30),秒(10))
scala在本例中无法推断类型的原因已在中解释,谢谢!更改后,程序给出了预期的结果,但同时也给出了另一个错误:java.util.NoSuchElementException:key not found:scala.collection.MapLike$class.default(MapLike.scala:228)的scala.collection.AbstractMap.default(Map.scala:58)的scala.collection.mutable.HashMap.apply(HashMap.scala:64)在org.apache.spark.stream.dstream.ReceiverInputDStream.getReceivedBlockInfo(ReceiverInputDStream.scala:77)我想知道这是怎么发生的?@user2895478我相信正是从这一点出发,问题在1.0.1和1.1.0中得到了解决