Apache kafka 卡夫卡反应器-如何禁用自动启动的卡夫卡消费者?
下面是我的卡夫卡消费者Apache kafka 卡夫卡反应器-如何禁用自动启动的卡夫卡消费者?,apache-kafka,spring-webflux,project-reactor,reactor,reactor-kafka,Apache Kafka,Spring Webflux,Project Reactor,Reactor,Reactor Kafka,下面是我的卡夫卡消费者 @Bean("kafkaConfluentInboundReceiver") @ConditionalOnProperty(value = "com.demo.kafka.core.inbound.confluent.topic-name", matchIfMissing = false) public KafkaReceiver<String, Object> kafkaInboundReceiver(
@Bean("kafkaConfluentInboundReceiver")
@ConditionalOnProperty(value = "com.demo.kafka.core.inbound.confluent.topic-name",
matchIfMissing = false)
public KafkaReceiver<String, Object> kafkaInboundReceiver() {
ReceiverOptions<String, Object> receiverOptions = ReceiverOptions.create(inboundConsumerConfigs());
receiverOptions.schedulerSupplier(() -> Schedulers
.fromExecutorService(applicationContext.getBean("inboundKafkaExecutorService", ExecutorService.class)));
receiverOptions.maxCommitAttempts(kafkaProperties.getKafka().getCore().getMaxCommitAttempts());
return KafkaReceiver.create(receiverOptions.addAssignListener(Collection::iterator)
.subscription(Collections.singleton(
kafkaProperties.getKafka()
.getCore().getInbound().getConfluent()
.getTopicName()))
.commitInterval(Duration.ZERO).commitBatchSize(0));
}
然而,我不确定如何在卡夫卡反应堆中实现(控制自动启动/停止行为)。我想要下面这样的东西
引入属性来处理自动启动/停止行为
@Value("${consumer.autostart:true}")
private boolean start;
使用上面的属性,我应该能够在卡夫卡反应器中设置卡夫卡自动启动标志,如下所示
factory.setAutoStartup(start);
return KafkaReceiver.create(receiverOptions.addAssignListener(Collection::iterator)
.subscription(Collections.singleton(
kafkaProperties.getKafka()
.getCore().getInbound().getConfluent()
.getTopicName()))
.commitInterval(Duration.ZERO).commitBatchSize(0)).setAutoStart(start);
注:.setAutoStart(启动)强>
这在卡夫卡反应堆中可行吗?如果可行,我该怎么做
更新:
protected void inboundEventHubListener(String topicName, List<String> allowedValues) {
Scheduler scheduler = Schedulers.fromExecutorService(kafkaExecutorService);
kafkaEventHubInboundReceiver
.receive()
.publishOn(scheduler)
.groupBy(receiverRecord -> {
try {
return receiverRecord.receiverOffset().topicPartition();
} catch (Throwable throwable) {
log.error("exception in groupby", throwable);
return Flux.empty();
}
}).flatMap(partitionFlux -> partitionFlux.publishOn(scheduler)
.map(record -> {
processMessage(record, topicName, allowedValues).block(
Duration.ofSeconds(60L));//This subscribe is to trigger processing of a message
return record;
}).concatMap(message -> {
log.info("Received message after processing offset: {} partition: {} ",
message.offset(), message.partition());
return message.receiverOffset()
.commit()
.onErrorContinue((t, o) -> log.error(
String.format("exception raised while commit offset %s", o), t)
);
})).onErrorContinue((t, o) -> {
try {
if (null != o) {
ReceiverRecord<String, Object> record = (ReceiverRecord<String, Object>) o;
ReceiverOffset offset = record.receiverOffset();
log.debug("failed to process message: {} partition: {} and message: {} ",
offset.offset(), record.partition(), record.value());
}
log.error(String.format("exception raised while processing message %s", o), t);
} catch (Throwable inner) {
log.error("encountered error in onErrorContinue", inner);
}
}).subscribeOn(scheduler).subscribe();
对于
反应堆卡夫卡
没有“自动启动”的概念;你完全在控制之中
在订阅从receiver.receive()
返回的Flux
之前,使用者不会“启动”
只需延迟
flux.subscribe()
,直到您准备好使用数据。谢谢您,Gary Russell。我已经用代码片段更新了问题,您能告诉我这是否可行吗?这是正确的,但对象将是flux
。
kafkaEventHubInboundReceiverObj = kafkaEventHubInboundReceiver.....subscribeOn(scheduler);
if(consumer.autostart) {
kafkaEventHubInboundReceiverObj.subscribe();
}