Apache flink 弗林克';s协处理函数不';不定时触发
我试着像那样把两条溪流聚合起来Apache flink 弗林克';s协处理函数不';不定时触发,apache-flink,flink-streaming,Apache Flink,Flink Streaming,我试着像那样把两条溪流聚合起来 val joinedStream = finishResultStream.keyBy(_.searchId) .connect(startResultStream.keyBy(_.searchId)) .process(new SomeCoProcessFunction) class SomeCoProcessFunction extends CoProcessFunction[SearchFinished, SearchCreated, Search
val joinedStream = finishResultStream.keyBy(_.searchId)
.connect(startResultStream.keyBy(_.searchId))
.process(new SomeCoProcessFunction)
class SomeCoProcessFunction extends CoProcessFunction[SearchFinished, SearchCreated, SearchAggregated] {
override def processElement1(finished: SearchFinished, ctx: CoProcessFunction[SearchFinished, SearchCreated, SearchAggregated]#Context, out: Collector[SearchAggregated]): Unit = {
// aggregating some "finished" data ...
}
override def processElement2(created: SearchCreated, ctx: CoProcessFunction[SearchFinished, SearchCreated, SearchAggregated]#Context, out: Collector[SearchAggregated]): Unit = {
val timerService = ctx.timerService()
timerService.registerEventTimeTimer(System.currentTimeMillis + 5000)
// aggregating some "created" data ...
}
override def onTimer(timestamp: Long, ctx: CoProcessFunction[SearchFinished, SearchCreated, SearchAggregated]#OnTimerContext, out: Collector[SearchAggregated]): Unit = {
val watermark: Long = ctx.timerService().currentWatermark()
println(s"watermark!!!! $watermark")
// clean up the state
}
然后在SomeCoProcessFunction
类中处理它们
val joinedStream = finishResultStream.keyBy(_.searchId)
.connect(startResultStream.keyBy(_.searchId))
.process(new SomeCoProcessFunction)
class SomeCoProcessFunction extends CoProcessFunction[SearchFinished, SearchCreated, SearchAggregated] {
override def processElement1(finished: SearchFinished, ctx: CoProcessFunction[SearchFinished, SearchCreated, SearchAggregated]#Context, out: Collector[SearchAggregated]): Unit = {
// aggregating some "finished" data ...
}
override def processElement2(created: SearchCreated, ctx: CoProcessFunction[SearchFinished, SearchCreated, SearchAggregated]#Context, out: Collector[SearchAggregated]): Unit = {
val timerService = ctx.timerService()
timerService.registerEventTimeTimer(System.currentTimeMillis + 5000)
// aggregating some "created" data ...
}
override def onTimer(timestamp: Long, ctx: CoProcessFunction[SearchFinished, SearchCreated, SearchAggregated]#OnTimerContext, out: Collector[SearchAggregated]): Unit = {
val watermark: Long = ctx.timerService().currentWatermark()
println(s"watermark!!!! $watermark")
// clean up the state
}
我想要的是在特定时间(5000毫秒)后清除状态,这就是必须使用的onTimer
。但因为它从未被解雇,我有点问自己我做错了什么
提前谢谢你的提示
更新:
解决方案是这样设置timeService(对费边·休斯克和贝克汉姆来说都是tnx):
timerService.RegisterProcessingTimer(timerService.currentProcessingTime()+5000)
我仍然没有真正弄清楚
timerService.registereventtimeter
做了什么,watermarkctx.timerService().currentWatermark()
无论EventTimer注册前多长时间,现在总是显示-9223372036854775808
。我看到您使用的是系统。currentTimeMillis
,它可能与Flink作业使用的时间特征(事件时间、处理时间、摄取时间)不同
尝试获取事件的时间戳ctx.timestamp()
,然后在其上添加5000ms。我看到您使用的是System.currentTimeMillis
,它可能不同于Flink作业使用的时间特征(事件时间、处理时间、摄取时间)
尝试获取事件的时间戳ctx.timestamp()
,然后在其上添加5000ms。问题是您正在使用处理时间戳(System.currentTimeMillis+5000
)注册事件时间计时器(timerService.registerEventTimeTimer
)
System.currentTimeMillis
返回当前机器时间,但事件时间不是基于机器时间,而是基于从水印计算的时间
您应该注册一个处理计时器,或者使用事件时间戳注册一个事件时间计时器。您可以从作为参数传递给processElement1()
和processElement2()
的Context
对象中获取当前水印的时间戳或当前记录的时间戳 问题在于,您正在使用处理时间戳(System.currentTimeMillis+5000
)注册事件时间计时器(timerService.registerEventTimeTimer
)
System.currentTimeMillis
返回当前机器时间,但事件时间不是基于机器时间,而是基于从水印计算的时间
您应该注册一个处理计时器,或者使用事件时间戳注册一个事件时间计时器。您可以从作为参数传递给processElement1()
和processElement2()
的Context
对象中获取当前水印的时间戳或当前记录的时间戳