Java 带窗口的KTable生成错误的类型
在Kafka中创建带有时间窗口的KTable时,我遇到了一些问题 我想创建一个统计流中ID数量的表,如下所示Java 带窗口的KTable生成错误的类型,java,apache-kafka,apache-kafka-streams,Java,Apache Kafka,Apache Kafka Streams,在Kafka中创建带有时间窗口的KTable时,我遇到了一些问题 我想创建一个统计流中ID数量的表,如下所示 ID (String) | Count (Long) X | 5 Y | 6 Z | 7 等等。我希望能够使用Kafka REST-API获取表,最好是.json 这是我目前的代码: StreamsBuilder builder = new StreamsBuilder();
ID (String) | Count (Long)
X | 5
Y | 6
Z | 7
等等。我希望能够使用Kafka REST-API获取表,最好是.json
这是我目前的代码:
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> streams = builder.stream(srcTopic);
KTable<Windowed<String>, Long> numCount = streams
.flatMapValues(value -> getID(value))
.groupBy((key, value) -> value)
.windowedBy(TimeWindows.of(windowSizeMs).advanceBy(advanceMs))
.count(Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("foo"));
它输出:
ID COUNT
2855 => ~
2857 => �
2859 => �
2861 => V(
2863 => �
2874 => �
2877 => J
2880 => �2
2891 => �=
无论哪种方式,我都不想使用KStream来收集输出,我想查询KTable。但是如前所述,我并不真正理解查询是如何工作的
更新
设法让它工作
ReadOnlyWindowStore<String, Long> windowStore =
kafkaStreams.store("tst", QueryableStoreTypes.windowStore());
long timeFrom = 0;
long timeTo = System.currentTimeMillis(); // now (in processing-time)
WindowStoreIterator<Long> iterator = windowStore.fetch("x", timeFrom, timeTo);
while (iterator.hasNext()) {
KeyValue<Long, Long> next = iterator.next();
long windowTimestamp = next.key;
System.out.println(windowTimestamp + ":" + next.value);
}
ReadOnlyWindows商店=
kafkaStreams.store(“tst”,QueryableStoreTypes.windowStore());
长时间间隔=0;
long-timeTo=System.currentTimeMillis();//现在(处理时间)
WindowsStoreIterator迭代器=WindowsStore.fetch(“x”,timeFrom,timeTo);
while(iterator.hasNext()){
KeyValue next=iterator.next();
长windowTimestamp=next.key;
System.out.println(windowTimestamp+“:”+next.value);
}
非常感谢,KTable的输出类型是
,因为在卡夫卡流中,多个窗口并行维护,以允许处理无序数据。因此,并非只有一个窗口实例,而是有多个并行的窗口实例。(参见)
保持“旧”窗口允许在数据延迟到达时更新它们。注意,Kafka Streams语义基于事件时间
您仍然可以查询KTable
——您只需要知道要查询的窗口
更新
JavaDoc描述了如何查询表:
KafkaStreams streams=…//数词
存储区queryableStoreName=…//queryableStoreName应该是具体化实例定义的存储的名称
ReadOnlyWindowStore localWindowStore=streams.store(queryableStoreName,QueryableStoreTypes.windowStore());
String key=“某个单词”;
长时间=。。。;
长时间=。。。;
WindowsStore迭代器countForWordsForWindows=LocalWindowsStore.fetch(key,timeFrom,timeTo);//密钥必须是本地的(应用程序状态在所有运行的Kafka Streams实例上共享)
好的,谢谢!我现在意识到,当然,它必须有多个窗口才能工作。问题仍然存在,它将KTable作为
返回。我用现在收到的输出编辑了我的问题。我不明白为什么角色是这样的,而不仅仅是“1”或“3”。。我是在打印窗口而不是第二个字符串吗?从您的代码中不清楚您是如何尝试查询的——我更新了我的答案。也许你应该考虑阅读文档。抱歉,我完全忘了用实际的查询来更新问题。我的错。谢谢你的帮助!还有一个问题,您知道有什么简单的方法可以获取WindowsStore中的所有键及其值吗?它似乎只适用于KeyValueStores…WindowedReadOnlyStore
自1.1版以来就有一个all()
方法(参见)
ReadOnlyWindowStore<String, Long> windowStore =
kafkaStreams.store("tst", QueryableStoreTypes.windowStore());
long timeFrom = 0;
long timeTo = System.currentTimeMillis(); // now (in processing-time)
WindowStoreIterator<Long> iterator = windowStore.fetch("x", timeFrom, timeTo);
while (iterator.hasNext()) {
KeyValue<Long, Long> next = iterator.next();
long windowTimestamp = next.key;
System.out.println(windowTimestamp + ":" + next.value);
}
KafkaStreams streams = ... // counting words
Store queryableStoreName = ... // the queryableStoreName should be the name of the store as defined by the Materialized instance
ReadOnlyWindowStore<String,Long> localWindowStore = streams.store(queryableStoreName, QueryableStoreTypes.<String, Long>windowStore());
String key = "some-word";
long fromTime = ...;
long toTime = ...;
WindowStoreIterator<Long> countForWordsForWindows = localWindowStore.fetch(key, timeFrom, timeTo); // key must be local (application state is shared over all running Kafka Streams instances)