Spring boot org.apache.kafka.common.KafkaException:无法构造kafka使用者
我手动启动Zookeeper,然后是Kafka服务器,最后是Kafka Rest服务器及其各自的属性文件。接下来,我将在tomcat上部署我的Spring Boot应用程序 在Tomcat日志跟踪中,我得到错误Spring boot org.apache.kafka.common.KafkaException:无法构造kafka使用者,spring-boot,apache-kafka,kafka-consumer-api,spring-kafka,Spring Boot,Apache Kafka,Kafka Consumer Api,Spring Kafka,我手动启动Zookeeper,然后是Kafka服务器,最后是Kafka Rest服务器及其各自的属性文件。接下来,我将在tomcat上部署我的Spring Boot应用程序 在Tomcat日志跟踪中,我得到错误org.springframework.context.ApplicationContextException:无法启动bean'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry';嵌套异常为org
org.springframework.context.ApplicationContextException:无法启动bean'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry';嵌套异常为org.apache.kafka.common.KafkaException:无法构造kafka使用者
,并且我的应用程序无法启动
错误日志
接收器配置类
消费性房地产
kafka-rest.properties
这些是我的类注释。他们看起来还好吗,还是我需要改变什么
kafka版本更新到0.10.1.1后出现新生成错误
我是否需要在ProducingChannelConfig、ConsumingChannelConfig以及InventoryReceiveConfig类中定义任何已添加为config的侦听器类
原因:java.lang.ClassNotFoundException:org.apache.kafka.common.ClusterResourceListener
您的类路径中缺少
kafka客户机
jar。您使用什么进行依赖关系管理?Maven和gradle应该自动将这个jar放在类路径上。您的kafka版本比它应该的版本旧,您有问题的类期望版本高于0.10.1.0,而您的版本低于0.10.0.1
只要升级你的卡夫卡罐子,它就会解决这个问题
在每个元数据响应之后,将有一次onUpdate(ClusterResource)调用。请注意,当Kafka broker版本低于0.10.1.0时,集群id可能为null
我在最近的一个项目中也遇到了这个错误。该错误也可能是由于缺少Kafka服务器造成的。应用程序正试图在application.properties文件中查找配置或服务器(PS:如果未指定应用程序应从哪个配置文件加载,则从application.properties加载应用程序) 解决方案 编辑配置(如果使用IntelliJ),并在program参数中,添加已添加Kafka引导服务器的配置文件。这样,当应用程序启动时,您已指定应用程序在包含Kafka引导服务器和详细信息的指定配置文件上运行
--spring.profiles.active= <profile_name>
--spring.profiles.active=
或者,如果您不想走这么长的路,只需创建application.properties文件而不指定任何配置文件,并将所有卡夫卡设置添加到此属性文件中。它应该可以正常工作,因为默认情况下,应用程序运行application.properties。在我的例子中,我在消费者中使用了KafkaAvroSerialiser作为反序列化程序,而不是kafkaavroderializer你的kafka jar版本是什么使用maven添加了pom.xml。添加了pom.xml。您的类路径上存在某种冲突版本;该类
ClusterResourceListener
已添加到0.10.2.x客户端中。也许你在tomcat的一个目录中有一个更新的版本?不要这样做(很多评论)删除它们并编辑问题。它与您的配置无关;这是一个类装入器问题。使用-verbose
JVM arg查看从哪些jar类加载。还有,你为什么要用这么旧的版本?SpringKafka的1.1.x行中的当前版本是1.1.7.Ok。即使在进行spring-kafka
和spring-kafka测试
至1.1.7.RELEASE
并将spring-integration-kafka
更改为2.1.2.RELEASE
至2.1.0.RELEASE
后,我仍在pom.xml中收到一个错误“缺少工件org.springframework.boot:spring-boot-starter-test:jar:1.5.9.RELEASE(点击查看更多信息)”虽然spring-boot-starter-test
已明确添加到我的pom.xmlI中,但我已将spring-kafka
和spring-kafka-test
升级为1.1.7。从1.1.1.1.1.1.1.1.1.RELEASE
发布,如评论中所述”@加里·拉塞尔“但是自动带到引用库
的依赖项是0.10.0.1
。我无法强制更改,因为它在pom.xml
中由于几个版本不兼容而产生错误。默认的卡夫卡jar作为依赖项带来的是kafka_2.11-0.10.0.1.jar
和kafka_2.11-0.10.0.1-test.jar
,只是为了在我的pom.xml
中对kafka-clients-0.10.0.1
进行注释之后添加这一点,带到类路径的默认依赖项也是kafka-clients-0.10.0.1.jar
和kafka-clients-0.10.0.1-test.jar
请查找我对“@Gary Russell”的最后一条评论。已成功将kafka JAR从0.10.0.1更新为0.10.1.1,但在构建过程中出现了新问题,在问题中添加。我认为您应该提出一个新问题,因为之后它会看起来一团糟
public class InventoryEventReceiver {
private static final Logger log = LoggerFactory.getLogger(InventoryEventReceiver.class);
private CountDownLatch latch = new CountDownLatch(1);
public CountDownLatch getLatch() {
return latch;
}
@KafkaListener(topics="inventory", containerFactory="kafkaListenerContainerFactory")
public void listenWithHeaders(
@Payload InventoryEvent event,
@Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
@Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) Integer key,
@Header(KafkaHeaders.RECEIVED_PARTITION_ID) int partition,
@Header(KafkaHeaders.OFFSET) String offset
) {
System.out.println("EVENT HAS BEEN RECEIVED by listenWithHeaders(InventoryEvent)");
System.out.println(event.toString());
log.info(System.currentTimeMillis() + "-- Received Event :\"" + event + "\" from partition:offset -- " + partition + ":" + offset +
" for topic : " + topic);
String urlForInventoryListeners = "http://localhost:8080/" + topic + "/listeners";
OutputStream os = null;
try {
URL objectUrl = new URL(urlForInventoryListeners);
HttpURLConnection con = (HttpURLConnection) objectUrl.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
con.setRequestProperty("topic", topic);
Gson gson = new Gson();
String eventJson = gson.toJson(event);
con.setDoOutput(true);
os = con.getOutputStream();
os.write(eventJson.getBytes("UTF-8"));
System.out.println("Event sent to " + objectUrl);
} catch (Exception e) {
e.printStackTrace();
System.out.println(e.getMessage());
} finally {
try {
os.close();
} catch (IOException e) {
e.printStackTrace();
System.out.println(e.getMessage());
}
}
latch.countDown();
}
}
@Configuration
@EnableKafka
public class InventoryReceiverConfig {
@Autowired
private KafkaConfig kafkaConfig;
@Bean
public static ConsumerFactory<String, InventoryEvent> consumerFactory() {
return new DefaultKafkaConsumerFactory<>(consumerConfigs(), new StringDeserializer(),
new JsonDeserializer<>(InventoryEvent.class));
}
@Bean
public static ConcurrentKafkaListenerContainerFactory<String, InventoryEvent> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, InventoryEvent> containerFactory = new ConcurrentKafkaListenerContainerFactory<>();
containerFactory.setConsumerFactory(consumerFactory());
containerFactory.setConcurrency(3);
containerFactory.getContainerProperties().setPollTimeout(3000);
return containerFactory;
}
@Bean
public static Map<String, Object> consumerConfigs() {
Map<String, Object> consumerProps = new HashMap<>();
consumerProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
consumerProps.put(ConsumerConfig.GROUP_ID_CONFIG,"inventory_consumers");
consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);
consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,JsonDeserializer.class);
consumerProps.put(ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG, "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor");
return consumerProps;
}
@Bean
public InventoryEventReceiver receiver() {
return new InventoryEventReceiver();
}
}
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
# Switch to enable topic deletion or not, default value is false
delete.topic.enable=true
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://localhost:9092
# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3
# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
##################### Confluent Proactive Support ######################
# If set to true, and confluent-support-metrics package is installed
# then the feature to collect and report support metrics
# ("Metrics") is enabled. If set to false, the feature is disabled.
#
confluent.support.metrics.enable=true
############################# Group Coordinator Settings #############################
# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0
# The customer ID under which support metrics will be collected and
# reported.
#
# When the customer ID is set to "anonymous" (the default), then only a
# reduced set of metrics is being collected and reported.
#
# Confluent customers
# -------------------
# If you are a Confluent customer, then you should replace the default
# value with your actual Confluent customer ID. Doing so will ensure
# that additional support metrics will be collected and reported.
#
confluent.support.customer.id=anonymous
# Zookeeper connection string
# comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002"
zookeeper.connect=127.0.0.1:2181
# timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
#consumer group id
group.id=test-consumer-group,inventory_consumers
#consumer timeout
#consumer.timeout.ms=5000
id=kafka-rest-test-server
schema.registry.url=http://localhost:8081
zookeeper.connect=localhost:2181
#
# Configure interceptor classes for sending consumer and producer metrics to Confluent Control Center
# Make sure that monitoring-interceptors-<version>.jar is on the Java class path
consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.psl.kafka.spring</groupId>
<artifactId>spring-kafka-webhook-service</artifactId>
<packaging>war</packaging>
<name>spring-kafka-webhook-service</name>
<description>Spring Kafka Webhook Service</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.9.RELEASE</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<repositories>
<repository>
<id>confluent</id>
<url>http://packages.confluent.io/maven/</url>
</repository>
</repositories>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
<org.springframework-version>5.0.0.RELEASE</org.springframework-version>
<org.springframework.security-version>4.0.1.RELEASE</org.springframework.security-version>
<org.aspectj-version>1.8.11</org.aspectj-version>
<org.slf4j-version>1.7.12</org.slf4j-version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
<version>1.1.1.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<version>1.1.1.RELEASE</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.2</version>
<exclusions>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.9.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.9.2</version>
</dependency>
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20160810</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.0</version>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-kafka</artifactId>
<version>2.1.0.RELEASE</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.0.1</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>monitoring-interceptors</artifactId>
<version>3.1.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
<version>0.0.1-SNAPSHOT</version>
</project>
@Configuration
public class InventorySenderConfig
@Configuration
@EnableKafka
public class InventoryReceiverConfig
@Component
public class KafkaConfig
@Configuration
public class ProducingChannelConfig
@Configuration
public class ConsumingChannelConfig
@RestController
public class KafkaWebhookController
@Service("webhookService")
public class KafkaServiceImpl
@EnableIntegration
@SpringBootApplication
@ComponentScan("com.psl.kafka")
public class SpringKafkaWebhookServiceApplication extends SpringBootServletInitializer
2017-12-26 13:11:44.490 INFO 13444 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka version : 0.10.1.1
2017-12-26 13:11:44.490 INFO 13444 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka commitId : f10ef2720b03b247
2017-12-26 13:12:44.499 ERROR 13444 --- [ main] o.s.k.support.LoggingProducerListener : Exception thrown when sending a message with key='inventory-events' and payload='Hello Spring Integration Kafka 0!' to topic inventory:
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
2017-12-26 13:12:44.501 WARN 13444 --- [ main] o.a.k.c.p.i.ProducerInterceptors : Error executing interceptor onAcknowledgement callback
java.lang.IllegalStateException: clusterResource is not defined
at io.confluent.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:174) ~[monitoring-interceptors-3.1.1.jar:na]
at io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor.onAcknowledgement(MonitoringProducerInterceptor.java:59) ~[monitoring-interceptors-3.1.1.jar:na]
at org.apache.kafka.clients.producer.internals.ProducerInterceptors.onSendError(ProducerInterceptors.java:116) ~[kafka-clients-0.10.1.1.jar:na]
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:489) [kafka-clients-0.10.1.1.jar:na]
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:436) [kafka-clients-0.10.1.1.jar:na]
--spring.profiles.active= <profile_name>