Spring boot StreamsException:无法初始化状态,如果Kafka流的多个实例在同一个状态目录中运行,则可能发生这种情况

Spring boot StreamsException:无法初始化状态,如果Kafka流的多个实例在同一个状态目录中运行,则可能发生这种情况,spring-boot,java-8,apache-kafka-streams,spring-kafka,Spring Boot,Java 8,Apache Kafka Streams,Spring Kafka,这是关于升级生产中的现有代码库,该代码库使用从kafka客户端、kafka streams、spring kafka 2.4.0到2.6.x的窗口,并将spring boot starter Parents从2.2.2.RELEASE升级到2.3.x,因为2.2与kafka streams 2.6不兼容 现有代码在旧版本(2.4.0,2.2 spring发行版)中包含以下提到的bean: 现在,在将kafka streams、kafka客户端升级到2.6.2和spring kafka升级到2.6

这是关于升级生产中的现有代码库,该代码库使用从kafka客户端、kafka streams、spring kafka 2.4.0到2.6.x的窗口,并将spring boot starter Parents从2.2.2.RELEASE升级到2.3.x,因为2.2与kafka streams 2.6不兼容

现有代码在旧版本(2.4.0,2.2 spring发行版)中包含以下提到的bean:

现在,在将kafka streams、kafka客户端升级到2.6.2和spring kafka升级到2.6.x之后,观察到以下异常:

2021-05-13 12:33:51.954 [Persistence-Realtime-Transformation] [main] WARN   o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'CustomFactoryBean'; nested exception is org.springframework.kafka.KafkaException: Could not start stream: ; nested exception is org.apache.kafka.streams.errors.StreamsException: Unable to initialize state, this can happen if multiple instances of Kafka Streams are running in the same state directory
   

这里的问题是,spring kafka的较新版本正在基于拓扑bean自动初始化kafka流的另一个实例,而generickafkaStreams的另一个bean正在从现有代码库初始化,这导致多个线程试图锁定状态目录,从而导致错误

即使在spring引导级别禁用KafkaAutoConfiguration也不会禁用此行为。这是一个如此痛苦的确定和损失了很多时间

修复方法是去掉拓扑bean,并拥有我们自己的定制kafka streams bean,代码如下:

  protected Topology customTopology()  {

    //topology code
    return streamsBuilder.build();
    }

    /**
     * This starts kafka stream application and sets the state listener and state
     * store listener.
     * 
     * @return KafkaStreams
     */
    @Bean("GenericKafkaStreams")
    public KafkaStreams kStream() {
    KafkaStreams kafkaStreams = new KafkaStreams(customTopology(), kstreamsconfigs);
    return kafkaStreams;
    }
  protected Topology customTopology()  {

    //topology code
    return streamsBuilder.build();
    }

    /**
     * This starts kafka stream application and sets the state listener and state
     * store listener.
     * 
     * @return KafkaStreams
     */
    @Bean("GenericKafkaStreams")
    public KafkaStreams kStream() {
    KafkaStreams kafkaStreams = new KafkaStreams(customTopology(), kstreamsconfigs);
    return kafkaStreams;
    }