Java 在KafkaStreams(0.11)中对聚合数据使用SessionWindows
我试图在Kafka(0.11)中的聚合函数中使用SessionWindows,但无法理解为什么会出现错误 以下是我的代码片段:Java 在KafkaStreams(0.11)中对聚合数据使用SessionWindows,java,aggregate,type-mismatch,apache-kafka-streams,Java,Aggregate,Type Mismatch,Apache Kafka Streams,我试图在Kafka(0.11)中的聚合函数中使用SessionWindows,但无法理解为什么会出现错误 以下是我的代码片段: // defining some values: public static final Integer SESSION_TIMEOUT_MS = 6000000; public static final String INTOPIC = "input"; public static final String HOST = "host"; // setting up
// defining some values:
public static final Integer SESSION_TIMEOUT_MS = 6000000;
public static final String INTOPIC = "input";
public static final String HOST = "host";
// setting up serdes:
final Serializer<JsonNode> jsonSerializer = new JsonSerializer();
final Deserializer<JsonNode> jsonDeserializer = new JsonDeserializer();
final Serde<JsonNode> jsonSerde = Serdes.serdeFrom(jsonSerializer, jsonDeserializer);
// some more code to build up the streams
KStreamBuilder builder = new KStreamBuilder();
KStream<String, JsonNode> dataStream = builder.stream(Serdes.String(), jsonSerde, INTOPIC);
// constructing the initalMessage ObjectNode:
ObjectNode initialMessage = JsonNodeFactory.instance.objectNode();
initialMessage.put("count", 0);
initialMessage.put("endTime", "");
// transforming data to KGroupedStream<String,JsonNode>
KGroupedStream<String, JsonNode> data = dataStream.map((key, value) ->{return new KeyValue<>(value.get(HOST).asText(), value); }).groupByKey(Serdes.String(), jsonSerde);
// finally aggregate the data usind SessionWindows
KTable<Windowed<String>, JsonNode> aggregatedData = data.aggregate(
() -> initialMessage,
(key, incomingMessage, initialMessage) -> countData(incomingMessage, initialMessage),
SessionWindows.with(SESSION_TIMEOUT_MS),
jsonSerde,
"aggregated-data");
private static JsonNode countData(JsonNode incomingMessage, JsonNode initialMessage){
// some dataprocessing
}
从聚合函数来看,一切正常
如果我不知道,eclipse会告诉我
KTable<Windowed<String>, JsonNode> aggregatedData = data.aggregate( [...])
类型不匹配:无法从ObjectNode转换为VR
以及:
DataWindowed类型中的方法countData(JsonNode,JsonNode)不适用于参数(JsonNode,VR)
我真的不明白,那些类型在哪里丢失了!
任何暗示都很好
Thx:D我确实需要实施合并:
Merger<? super String, JsonNode>tmpMerger = new MergerClass<String, JsonNode>();
MergerIs它只是一个输入错误(key.value)
->(key,value)
(逗号而不是点)?是的,对不起。刚刚在这篇文章中修复了它。但这并不是解决办法。你有没有其他的想法,如何解决这个问题?不是从代码开始的。也许我们的示例repo有帮助:我们有几个使用lambdas的示例。感谢您的回复,@MatthiasJ.Sax我查看了这些示例,但没有发现任何新的内容,除了SessionWindows在count方法中工作良好之外。不过我意识到,当使用SessionWindows时,聚合需要合并!!哦!
SessionWindows.with(SESSION_TIMEOUT_MS)
KTable<Windowed<String>, JsonNode> aggregatedData = data.aggregate( [...])
() -> initialMessage
(key, incomingMessage, initialMessage) -> countData(incomingMessage, initialMessage),
Merger<? super String, JsonNode>tmpMerger = new MergerClass<String, JsonNode>();
KTable<Windowed<String>, JsonNode> aggregatedData = data.aggregate(
() -> initialMessage,
(key, incomingMessage, initialMessage) -> countData(incomingMessage, initialMessage),
tmpMerger,
SessionWindows.with(SESSION_TIMEOUT_MS),
jsonSerde,
"aggregated-data");