Spring 添加多个KafkalistenerContainerFactory时出现问题
嗨,我目前正在涉猎春季卡夫卡,并成功地为我的听众添加了一个卡夫卡ListenerContainerFactory。现在,我想添加多个KafkalistenerContainerFactory(一个用于将包含json消息的主题,另一个用于字符串)。见下面的代码:Spring 添加多个KafkalistenerContainerFactory时出现问题,spring,spring-boot,apache-kafka,spring-kafka,Spring,Spring Boot,Apache Kafka,Spring Kafka,嗨,我目前正在涉猎春季卡夫卡,并成功地为我的听众添加了一个卡夫卡ListenerContainerFactory。现在,我想添加多个KafkalistenerContainerFactory(一个用于将包含json消息的主题,另一个用于字符串)。见下面的代码: @EnableKafka @Configuration public class KafkaConsumersConfig { private final KafkaConfiguration kafkaConfiguratio
@EnableKafka
@Configuration
public class KafkaConsumersConfig {
private final KafkaConfiguration kafkaConfiguration;
@Autowired
public KafkaConsumersConfig(KafkaConfiguration kafkaConfiguration) {
this.kafkaConfiguration = kafkaConfiguration;
}
@Bean
public KafkaListenerContainerFactory<?> kafkaJsonListenerContainerFactory(){
ConcurrentKafkaListenerContainerFactory<String,Record> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(jsonConsumerFactory());
factory.setConcurrency(3);
factory.setAutoStartup(true);
return factory;
}
@Bean
public ConsumerFactory<String,Record> jsonConsumerFactory(){
JsonDeserializer<Record> jsonDeserializer = new JsonDeserializer<>(Record.class);
return new DefaultKafkaConsumerFactory<>(jsonConsumerConfigs(),new StringDeserializer(), jsonDeserializer);
}
@Bean
public Map<String,Object> jsonConsumerConfigs(){
Map<String,Object> propsMap = new HashMap<>();
propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaConfiguration.getBrokerAddress());
propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, kafkaConfiguration.getJsonGroupId());
propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, kafkaConfiguration.getAutoCommit());
propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, kafkaConfiguration.getAutoCommitInterval());
propsMap.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, kafkaConfiguration.getSessionTimeout());
return propsMap;
}
@Bean
public KafkaListenerContainerFactory<?> kafkaFileListenerContainerFactory(){
ConcurrentKafkaListenerContainerFactory<String,String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(fileConsumerFactory());
factory.setConcurrency(3);
factory.setAutoStartup(true);
return factory;
}
@Bean
public ConsumerFactory<String,String> fileConsumerFactory(){
return new DefaultKafkaConsumerFactory<>(fileConsumerConfigs());
}
@Bean
public Map<String,Object> fileConsumerConfigs(){
Map<String,Object> propsMap = new HashMap<>();
propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaConfiguration.getBrokerAddress());
propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, kafkaConfiguration.getFileGroupId());
propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, kafkaConfiguration.getAutoCommit());
propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, kafkaConfiguration.getAutoCommitInterval());
propsMap.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, kafkaConfiguration.getSessionTimeout());
return propsMap;
}
}
我做错了什么?看起来你不会依赖于弹簧靴的功能 Spring Boot在KafkaAutoConfiguration中提供:
@Bean
@ConditionalOnMissingBean(ConsumerFactory.class)
public ConsumerFactory<?, ?> kafkaConsumerFactory() {
因为您的ConsumerFactory
bean不是ConsumerFactory
类型
因此:
- 只需将以下内容添加到应用程序属性文件中,即可从Spring Boot自动配置中排除
:KafkaAutoConfiguration
spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration
- 或者将其中一个
bean重命名为KafkaListenerContainerFactory
,以在启动时覆盖它KafkaListenerContainerFactory
- 或者将一个
bean作为ConsumerFactory
类型ConsumerFactory
@KafkaListener(topics = "fileTopic", containerFactory = "kafkaFileListenerContainerFactory")
public void fileConsumer(...) {...}
@KafkaListener(topics = "jsonTopic", containerFactory = "kafkaJsonListenerContainerFactory")
public void jsonConsumer(...) {...}
我在代码下面实现了它,它对我来说运行良好。
// LISTENER 1
@Bean
@ConditionalOnMissingBean(name = "yourListenerFactory1")
public ConsumerFactory<String, YourCustomObject1> yourConsumerFactory1() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "YOUR-GROUP-1");
return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(),
new JsonDeserializer<>(YourCustomObject1.class));
}
@Bean(name = "yourListenerFactory1")
public ConcurrentKafkaListenerContainerFactory<String, YourCustomObject1>
yourListenerFactory1() {
ConcurrentKafkaListenerContainerFactory<String, YourCustomObject1> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(yourConsumerFactory1());
ContainerProperties containerProperties = factory.getContainerProperties();
containerProperties.setPollTimeout(...);
containerProperties.setAckMode(AckMode...);
return factory;
}
// LISTENER 2
@Bean
@ConditionalOnMissingBean(name = "yourListenerFactory2")
public ConsumerFactory<String, YourCustomObject2> yourConsumerFactory2() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "YOUR-GROUP-2");
return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(),
new JsonDeserializer<>(YourCustomObject2.class));
}
@Bean(name = "yourListenerFactory2")
public ConcurrentKafkaListenerContainerFactory<String, YourCustomObject2>
yourListenerFactory2() {
ConcurrentKafkaListenerContainerFactory<String, YourCustomObject2> factory
= new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(yourConsumerFactory2());
ContainerProperties containerProperties = factory.getContainerProperties();
containerProperties.setPollTimeout(...);
containerProperties.setAckMode(AckMode...);
return factory;
}
消费者2
@KafkaListener(id = "your-cousumer-2",
topicPattern = "your-topic-2",
containerFactory = "yourListenerFactory2")
public void consumer2(YourCustomObject2 data,
Acknowledgment acknowledgment,
@Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitions,
@Header(KafkaHeaders.RECEIVED_TOPIC) List<String> topics,
@Header(KafkaHeaders.OFFSET) List<Long> offsets) throws Exception { ... }
@KafkaListener(id=“your-cousumer-2”,
topicPattern=“your-topic-2”,
containerFactory=“yourListenerFactory2”)
public void consumer2(您的CustomObject2数据,
致谢致谢致谢,
@标题(KafkaHeaders.RECEIVED_PARTITION_ID)列出分区,
@标题(KafkaHeaders.RECEIVED_主题)列出主题,
@标头(KafkaHeaders.OFFSET)列表偏移量)引发异常{…}
另外,我的卡夫卡模板是
@Autowired
KafkaTemplate<String, Object> kafkaTemplate;
@Autowired
卡夫卡坦普尔卡夫卡坦普尔;
如果您对不同的主题使用相同的组id,这是否有效?我没有尝试过。我在同一组中使用过多个主题。它可以工作。我想这只是因为你关闭了KafkaAutoConfiguration
?您使用@ConditionalOnMissingBean(name=“yourListenerFactory1”)
所做的事情似乎没有多大意义,因为只有当您的ConcurrentKafkListenerContainerFactory
不存在时,您才启用ConsumerFactory。您不应该在那里通过KafkListenerContainerFactory而不是ConsumerFactory吗?别这么想works@Rodrigo你是对的。我写错了。我已经编辑了我的答案。谢谢
@KafkaListener(id = "your-cousumer-1",
topicPattern = "your-topic-1",
containerFactory = "yourListenerFactory1")
public void consumer1(YourCustomObject1 data,
Acknowledgment acknowledgment,
@Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitions,
@Header(KafkaHeaders.RECEIVED_TOPIC) List<String> topics,
@Header(KafkaHeaders.OFFSET) List<Long> offsets) throws Exception { ... }
@KafkaListener(id = "your-cousumer-2",
topicPattern = "your-topic-2",
containerFactory = "yourListenerFactory2")
public void consumer2(YourCustomObject2 data,
Acknowledgment acknowledgment,
@Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitions,
@Header(KafkaHeaders.RECEIVED_TOPIC) List<String> topics,
@Header(KafkaHeaders.OFFSET) List<Long> offsets) throws Exception { ... }
@Autowired
KafkaTemplate<String, Object> kafkaTemplate;