Spring boot StreamsException:无法初始化状态,如果Kafka流的多个实例在同一个状态目录中运行,则可能发生这种情况
这是关于升级生产中的现有代码库,该代码库使用从kafka客户端、kafka streams、spring kafka 2.4.0到2.6.x的窗口,并将spring boot starter Parents从2.2.2.RELEASE升级到2.3.x,因为2.2与kafka streams 2.6不兼容 现有代码在旧版本(2.4.0,2.2 spring发行版)中包含以下提到的bean: 现在,在将kafka streams、kafka客户端升级到2.6.2和spring kafka升级到2.6.x之后,观察到以下异常:Spring boot StreamsException:无法初始化状态,如果Kafka流的多个实例在同一个状态目录中运行,则可能发生这种情况,spring-boot,java-8,apache-kafka-streams,spring-kafka,Spring Boot,Java 8,Apache Kafka Streams,Spring Kafka,这是关于升级生产中的现有代码库,该代码库使用从kafka客户端、kafka streams、spring kafka 2.4.0到2.6.x的窗口,并将spring boot starter Parents从2.2.2.RELEASE升级到2.3.x,因为2.2与kafka streams 2.6不兼容 现有代码在旧版本(2.4.0,2.2 spring发行版)中包含以下提到的bean: 现在,在将kafka streams、kafka客户端升级到2.6.2和spring kafka升级到2.6
2021-05-13 12:33:51.954 [Persistence-Realtime-Transformation] [main] WARN o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'CustomFactoryBean'; nested exception is org.springframework.kafka.KafkaException: Could not start stream: ; nested exception is org.apache.kafka.streams.errors.StreamsException: Unable to initialize state, this can happen if multiple instances of Kafka Streams are running in the same state directory
这里的问题是,spring kafka的较新版本正在基于拓扑bean自动初始化kafka流的另一个实例,而generickafkaStreams的另一个bean正在从现有代码库初始化,这导致多个线程试图锁定状态目录,从而导致错误 即使在spring引导级别禁用KafkaAutoConfiguration也不会禁用此行为。这是一个如此痛苦的确定和损失了很多时间 修复方法是去掉拓扑bean,并拥有我们自己的定制kafka streams bean,代码如下:
protected Topology customTopology() {
//topology code
return streamsBuilder.build();
}
/**
* This starts kafka stream application and sets the state listener and state
* store listener.
*
* @return KafkaStreams
*/
@Bean("GenericKafkaStreams")
public KafkaStreams kStream() {
KafkaStreams kafkaStreams = new KafkaStreams(customTopology(), kstreamsconfigs);
return kafkaStreams;
}
protected Topology customTopology() {
//topology code
return streamsBuilder.build();
}
/**
* This starts kafka stream application and sets the state listener and state
* store listener.
*
* @return KafkaStreams
*/
@Bean("GenericKafkaStreams")
public KafkaStreams kStream() {
KafkaStreams kafkaStreams = new KafkaStreams(customTopology(), kstreamsconfigs);
return kafkaStreams;
}