Java Kafka流聚合引发异常
这是我的Kafka streams代码,它使用滑动窗口对时间窗口中的所有整数数据求和Java Kafka流聚合引发异常,java,serialization,apache-kafka,apache-kafka-streams,Java,Serialization,Apache Kafka,Apache Kafka Streams,这是我的Kafka streams代码,它使用滑动窗口对时间窗口中的所有整数数据求和 public class KafkaWindowingLIS { public static void main(String[] args) { Properties config = new Properties(); config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafkahost:9092");
public class KafkaWindowingLIS {
public static void main(String[] args) {
Properties config = new Properties();
config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafkahost:9092");
config.put(StreamsConfig.APPLICATION_ID_CONFIG, "kafka-windowing-lis");
config.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
config.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
Integer uid = 1;
long tenSeconds = 1000 * 10;
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> dataStream = builder.stream("kafka-windowing-lis");
KStream<Integer, Integer> integerKStream = dataStream
.filter((key, val) -> { //Filter only numbers from Stream
try {
Integer.parseInt(val);
return true;
} catch (NumberFormatException e) {
return false;
}
})
.map((key, val) -> new KeyValue<Integer, Integer>(uid, Integer.parseInt(val)));
TimeWindowedKStream<Integer, Integer> timeWindowedKStream = integerKStream
.groupBy((k, v) -> k, Serialized.with(Serdes.Integer(), Serdes.Integer())) //Killed my time
.windowedBy(TimeWindows.of(tenSeconds));
timeWindowedKStream.aggregate(
() -> 0,
(key, value, aggregate) -> value + aggregate)
.toStream().print(Printed.toSysOut());
KafkaStreams kafkaStreams = new KafkaStreams(builder.build(), config);
// kafkaStreams.cleanUp();
kafkaStreams.start();
Runtime.getRuntime().addShutdownHook(new Thread(kafkaStreams::close));
}
}
我在不同的页面中寻找解决方案,看起来我缺少了一些东西。
任何投入都将受到极大欢迎。
谢谢。因为您使用了
aggregate()
,所以需要通过aggregate(…,Materialized.with(…)
显式设置输出值serde。输出值类型可能与输入值类型不同,因此输入值serde不能重复使用。(由于Java类型擦除,Kafka Streams不知道该类型实际上没有更改…)因此,Kafka Streams从配置返回默认serde
或者,您可以使用
reduce()
而不是aggregate
来解决问题。在reduce()
中,输出类型与输入类型相同,因此输入值serde可用于输出值。尝试将默认serdes更改为整数serdes,并仅在读取输入时显式设置字符串serdestopic@Anthony,如果同时更改了序列化器/反序列化器或SERDE,最好更改您的组名-application.id。您的应用程序可能使用一个序列化程序序列化消息,然后尝试使用不兼容的反序列化程序进行读取。这有帮助。我可以知道在哪里可以找到学习卡夫卡流的标准文档/资源吗?有很多资源:官方的Apache卡夫卡文档:合流文档:--这些应该是最好的资源。还有一个非常有用的示例存储库:
Caused by: org.apache.kafka.streams.errors.StreamsException: A serializer (org.apache.kafka.common.serialization.StringSerializer) is not compatible to the actual value type (value type: java.lang.Integer). Change the default Serdes in StreamConfig or provide correct Serdes via method parameters.
at org.apache.kafka.streams.state.StateSerdes.rawValue(StateSerdes.java:204)
at org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:138)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl$WindowStoreReadWriteDecorator.put(ProcessorContextImpl.java:533)
at org.apache.kafka.streams.kstream.internals.KStreamWindowAggregate$KStreamWindowAggregateProcessor.process(KStreamWindowAggregate.java:138)
at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:117)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:201)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:180)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:133)
at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:87)
at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:366)
... 5 more
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String
at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:28)
at org.apache.kafka.streams.state.internals.ValueAndTimestampSerializer.serialize(ValueAndTimestampSerializer.java:59)
at org.apache.kafka.streams.state.internals.ValueAndTimestampSerializer.serialize(ValueAndTimestampSerializer.java:50)
at org.apache.kafka.streams.state.internals.ValueAndTimestampSerializer.serialize(ValueAndTimestampSerializer.java:27)
at org.apache.kafka.streams.state.StateSerdes.rawValue(StateSerdes.java:192)
... 14 more