Scala Kafka Streams PAPI:处理器关闭在启动时被调用,而且调用频率太高
如中所示,您应该关闭处理器中使用的存储,并覆盖close方法 在WordCountProcessor示例中,它显示了如何在调用close()方法中关闭存储 我也做过类似的事情(我没有在init()方法中启动它们,而是在Scala中使用lazy val),我发现我的Processor close()方法在创建存储之后被多次调用Scala Kafka Streams PAPI:处理器关闭在启动时被调用,而且调用频率太高,scala,apache-kafka,apache-kafka-streams,Scala,Apache Kafka,Apache Kafka Streams,如中所示,您应该关闭处理器中使用的存储,并覆盖close方法 在WordCountProcessor示例中,它显示了如何在调用close()方法中关闭存储 我也做过类似的事情(我没有在init()方法中启动它们,而是在Scala中使用lazy val),我发现我的Processor close()方法在创建存储之后被多次调用 class EventWindowProcessor(sessionStoreName: String, lastSessionByChannelStoreName: St
class EventWindowProcessor(sessionStoreName: String, lastSessionByChannelStoreName: String, lastChannelStoreName: String)
extends AbstractProcesso
// example of a store
private lazy val lastChannelStore: KeyValueStore[MyKey, Channel] =
context()
.getStateStore(lastChannelStoreName)
.asInstanceOf[KeyValueStore[MyKey, Channel]]
override def init(context: ProcessorContext) = {
super.init(context)
}
override def close() = {
logger.info("CLOSING PROCESSOR")
}
override def process(key: String, value: String): Unit = {
// ... my stuff here
}
因此,我得到了以下输出,显示了processor.close()在拓扑运行开始时被调用了很多次,并且在应用程序的后期也被调用
[2018-06-08 05:13:16,255] INFO Stream Application starting, name: stream-processor (my.package.StreamProcessorApplication$)
[2018-06-08 05:13:16,760] INFO Topology: Sub-topologies:
Sub-topology: 0
Source: event-source (topics: [events])
--> session-processor
Processor: session-processor (stores: [sessionStoreName, lastSessionByChannelStoreName, lastChannelStoreName])
--> error-event-sink, order-sink, pageviews-sink, session-sink
<-- event-source
Sink: error-event-sink (topic: error-events)
<-- session-processor
Sink: order-sink (topic: orders)
<-- session-processor
Sink: pageviews-sink (topic: pageviews)
<-- session-processor
Sink: session-sink (topic: sessions)
<-- session-processor
Global Stores:
none
(my.package.StreamProcessorApplication$)
[2018-06-08 05:14:01,425] INFO CLOSING PROCESSOR (my.package.StreamProcessor)
[2018-06-08 05:14:01,539] INFO CLOSING PROCESSOR (my.package.StreamProcessor)
[2018-06-08 05:14:01,640] INFO CLOSING PROCESSOR (my.package.StreamProcessor)
... (102 lines like that)
[2018-06-08 05:29:05,548] INFO .... my own application logging here
[2018-06-08 05:13:16255]信息流应用程序启动,名称:流处理器(my.package.StreamProcessorApplication$)
[2018-06-08 05:13:16760]信息拓扑:子拓扑:
子拓扑:0
来源:事件源(主题:[事件])
-->会话处理器
处理器:会话处理器(存储:[sessionStoreName,lastSessionByChannelStoreName,lastChannelStoreName])
-->错误事件接收器、订单接收器、页面视图接收器、会话接收器
文档中的示例不正确。您不应该关闭该存储--该存储由Kafka Streams管理,Kafka Streams将为您的用户关闭该存储。(我将做一个PR来修复代码示例。感谢您指出。)
关于调用处理器#close()
:处理器可能会关闭并重新打开。这发生在再平衡期间。因此,编写代码的方式必须确保它能够正确地用于多次调用init()
和close()
——我们最近为此更新了JavaDocs(改进的JavaDocs将成为Kafka 2.0版本的一部分)