Apache kafka 从架构注册表获取409,同时将记录保存到本地状态存储,其中多个状态存储与单个处理器关联 长篇短文:我正在实现处理器拓扑结构:处理器将接收到的记录存储到相应的本地状态存储区,并在记录到达时进行基于事件的处理。相关代码如下所示: @Override public void configureBuilder(StreamsBuilder builder) { final Map<String, String> serdeConfig = Collections.singletonMap("schema.registry.url", processorConfig.getSchemaRegistryUrl()); final Serde<GenericRecord> valueSerde = new GenericAvroSerde(); valueSerde.configure(serdeConfig, false); // `true` for record keys final Serde<EventKey> keySerde = new SpecificAvroSerde(); keySerde.configure(serdeConfig, true); // `true` for record keys Map<String, String> stateStoreConfigMap = new HashMap<>(); //stateStoreConfigMap.put(KafkaAvroSerializerConfig.VALUE_SUBJECT_NAME_STRATEGY, RecordNameStrategy.class.getName()); StoreBuilder<KeyValueStore<EventKey, GenericRecord>> aggSequenceStateStoreBuilder = Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(processStateStore), keySerde, valueSerde) .withLoggingEnabled(stateStoreConfigMap) .withCachingEnabled(); final Serde<EnrichedSmcHeatData> enrichedSmcHeatDataSerde = new SpecificAvroSerde<>(); enrichedSmcHeatDataSerde.configure(serdeConfig, false); // `true` for record keys StoreBuilder<KeyValueStore<EventKey, EnrichedSmcHeatData>> enrichedSmcHeatStateStoreBuilder = Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore("enriched-smc-heat-state-store"), keySerde, enrichedSmcHeatDataSerde) .withLoggingEnabled(stateStoreConfigMap) .withCachingEnabled(); Topology topology = builder.build(); topology .addSource( PROCESS_EVENTS_SOURCE, keySerde.deserializer(), valueSerde.deserializer(), processorConfig.getInputCcmProcessEvents()) .addSource( SCHEDULED_SEQUENCES_SOURCE, keySerde.deserializer(), valueSerde.deserializer(), processorConfig.getScheduledCastSequences()) .addSource( SMC_HEAT_EVENTS_SOURCE, keySerde.deserializer(), valueSerde.deserializer(), processorConfig.getInputSmcHeatEvents()) .addProcessor( PROCESS_STATE_AGGREGATOR, () -> new ProcessStateProcessor(processStateStore, processorConfig), PROCESS_EVENTS_SOURCE, SCHEDULED_SEQUENCES_SOURCE, SMC_HEAT_EVENTS_SOURCE) .addStateStore(aggSequenceStateStoreBuilder, PROCESS_STATE_AGGREGATOR) .addStateStore(enrichedSmcHeatStateStoreBuilder, PROCESS_STATE_AGGREGATOR); @覆盖 公共无效配置生成器(StreamsBuilder生成器){ 最终地图解编图= Collections.singletonMap(“schema.registry.url”,processorConfig.getSchemaRegistryUrl()); 最终Serde值Serde=新的通用VROSERDE(); valueSerde.configure(serdeConfig,false);//`true`用于记录键 最终Serde keySerde=新的SpecificAvroSerde(); keySerde.configure(serdeConfig,true);//`true`用于记录键 Map stateStoreConfigMap=新建HashMap(); //stateStoreConfigMap.put(KafkaAvroSerializerConfig.VALUE\u SUBJECT\u NAME\u策略,RecordNameStrategy.class.getName()); StoreBuilder aggSequenceStateStoreBuilder= Stores.keyValueStoreBuilder( persistentKeyValueStore(processStateStore)、keySerde、valueSerde) .withLoggingEnabled(stateStoreConfigMap) .用咯咯笑的方式(); 最终Serde enrichedSmcHeatDataSerde=新的SpecificAvroSerde(); enrichedSmcHeatDataSerde.configure(serdeConfig,false);//'true'用于记录键 StoreBuilder EnrichedSmartStateStoreBuilder= Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(“浓缩smc热状态存储”),keySerde,EnrichedSMC热状态存储) .withLoggingEnabled(stateStoreConfigMap) .用咯咯笑的方式(); 拓扑=builder.build(); 拓扑学 .addSource( 进程\事件\源, keySerde.deserializer(), valueSerde.deserializer(), processorConfig.GetInputCmProcessEvents()) .addSource( 计划的\u序列\u源, keySerde.deserializer(), valueSerde.deserializer(), processorConfig.getScheduledCastSequences()) .addSource( SMC\u热事件\u源, keySerde.deserializer(), valueSerde.deserializer(), processorConfig.getInputSmcHeatEvents()) .addProcessor( 进程状态聚合器, ()->新的ProcessStateProcessor(processStateStore、processorConfig), 进程\事件\源, 计划的\u序列\u源, SMC(热事件源) .addStateStore(aggSequenceStateStoreBuilder、PROCESS\u STATE\u聚合器) .addStateStore(enrichedSmcHeatStateStoreBuilder、进程状态聚合器);

Apache kafka 从架构注册表获取409,同时将记录保存到本地状态存储,其中多个状态存储与单个处理器关联 长篇短文:我正在实现处理器拓扑结构:处理器将接收到的记录存储到相应的本地状态存储区,并在记录到达时进行基于事件的处理。相关代码如下所示: @Override public void configureBuilder(StreamsBuilder builder) { final Map<String, String> serdeConfig = Collections.singletonMap("schema.registry.url", processorConfig.getSchemaRegistryUrl()); final Serde<GenericRecord> valueSerde = new GenericAvroSerde(); valueSerde.configure(serdeConfig, false); // `true` for record keys final Serde<EventKey> keySerde = new SpecificAvroSerde(); keySerde.configure(serdeConfig, true); // `true` for record keys Map<String, String> stateStoreConfigMap = new HashMap<>(); //stateStoreConfigMap.put(KafkaAvroSerializerConfig.VALUE_SUBJECT_NAME_STRATEGY, RecordNameStrategy.class.getName()); StoreBuilder<KeyValueStore<EventKey, GenericRecord>> aggSequenceStateStoreBuilder = Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(processStateStore), keySerde, valueSerde) .withLoggingEnabled(stateStoreConfigMap) .withCachingEnabled(); final Serde<EnrichedSmcHeatData> enrichedSmcHeatDataSerde = new SpecificAvroSerde<>(); enrichedSmcHeatDataSerde.configure(serdeConfig, false); // `true` for record keys StoreBuilder<KeyValueStore<EventKey, EnrichedSmcHeatData>> enrichedSmcHeatStateStoreBuilder = Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore("enriched-smc-heat-state-store"), keySerde, enrichedSmcHeatDataSerde) .withLoggingEnabled(stateStoreConfigMap) .withCachingEnabled(); Topology topology = builder.build(); topology .addSource( PROCESS_EVENTS_SOURCE, keySerde.deserializer(), valueSerde.deserializer(), processorConfig.getInputCcmProcessEvents()) .addSource( SCHEDULED_SEQUENCES_SOURCE, keySerde.deserializer(), valueSerde.deserializer(), processorConfig.getScheduledCastSequences()) .addSource( SMC_HEAT_EVENTS_SOURCE, keySerde.deserializer(), valueSerde.deserializer(), processorConfig.getInputSmcHeatEvents()) .addProcessor( PROCESS_STATE_AGGREGATOR, () -> new ProcessStateProcessor(processStateStore, processorConfig), PROCESS_EVENTS_SOURCE, SCHEDULED_SEQUENCES_SOURCE, SMC_HEAT_EVENTS_SOURCE) .addStateStore(aggSequenceStateStoreBuilder, PROCESS_STATE_AGGREGATOR) .addStateStore(enrichedSmcHeatStateStoreBuilder, PROCESS_STATE_AGGREGATOR); @覆盖 公共无效配置生成器(StreamsBuilder生成器){ 最终地图解编图= Collections.singletonMap(“schema.registry.url”,processorConfig.getSchemaRegistryUrl()); 最终Serde值Serde=新的通用VROSERDE(); valueSerde.configure(serdeConfig,false);//`true`用于记录键 最终Serde keySerde=新的SpecificAvroSerde(); keySerde.configure(serdeConfig,true);//`true`用于记录键 Map stateStoreConfigMap=新建HashMap(); //stateStoreConfigMap.put(KafkaAvroSerializerConfig.VALUE\u SUBJECT\u NAME\u策略,RecordNameStrategy.class.getName()); StoreBuilder aggSequenceStateStoreBuilder= Stores.keyValueStoreBuilder( persistentKeyValueStore(processStateStore)、keySerde、valueSerde) .withLoggingEnabled(stateStoreConfigMap) .用咯咯笑的方式(); 最终Serde enrichedSmcHeatDataSerde=新的SpecificAvroSerde(); enrichedSmcHeatDataSerde.configure(serdeConfig,false);//'true'用于记录键 StoreBuilder EnrichedSmartStateStoreBuilder= Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(“浓缩smc热状态存储”),keySerde,EnrichedSMC热状态存储) .withLoggingEnabled(stateStoreConfigMap) .用咯咯笑的方式(); 拓扑=builder.build(); 拓扑学 .addSource( 进程\事件\源, keySerde.deserializer(), valueSerde.deserializer(), processorConfig.GetInputCmProcessEvents()) .addSource( 计划的\u序列\u源, keySerde.deserializer(), valueSerde.deserializer(), processorConfig.getScheduledCastSequences()) .addSource( SMC\u热事件\u源, keySerde.deserializer(), valueSerde.deserializer(), processorConfig.getInputSmcHeatEvents()) .addProcessor( 进程状态聚合器, ()->新的ProcessStateProcessor(processStateStore、processorConfig), 进程\事件\源, 计划的\u序列\u源, SMC(热事件源) .addStateStore(aggSequenceStateStoreBuilder、PROCESS\u STATE\u聚合器) .addStateStore(enrichedSmcHeatStateStoreBuilder、进程状态聚合器);,apache-kafka,apache-kafka-streams,confluent-schema-registry,Apache Kafka,Apache Kafka Streams,Confluent Schema Registry,如果aggSequenceStateStoreBuilder创建的存储有更新,记录值可以毫无问题地保存到存储。但是,如果第二个存储有更新,则引发以下错误: 原因: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: 正在注册的架构与的早期架构不兼容 主题 “ccm进程事件处理器ccm进程状态存储更改日志值”; 错误代码:409 我的用例:状态处理器接受来自多个源主题的入站记录,并在记录从任何源

如果aggSequenceStateStoreBuilder创建的存储有更新,记录值可以毫无问题地保存到存储。但是,如果第二个存储有更新,则引发以下错误:

原因: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: 正在注册的架构与的早期架构不兼容 主题 “ccm进程事件处理器ccm进程状态存储更改日志值”; 错误代码:409

我的用例:状态处理器接受来自多个源主题的入站记录,并在记录从任何源主题到达时进行事件处理(包括将修改后的值存储到相应的存储)

对于同一个处理器,似乎只能有一个架构在架构注册表中注册。这是出于设计,还是我遗漏了什么,或者我有什么替代选项


提前感谢!

@Matthias J.Sax Hi Matthias,我一直很喜欢阅读您关于卡夫卡流主题的评论,希望您能在这里提供一些见解,谢谢。报告的问题是由于处理器#init方法中没有正确加载状态存储。干杯