Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/apache-kafka/3.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Apache kafka 卡夫卡溪流:“;更新的群集元数据版本“;跑了很长时间_Apache Kafka_Apache Kafka Streams - Fatal编程技术网

Apache kafka 卡夫卡溪流:“;更新的群集元数据版本“;跑了很长时间

Apache kafka 卡夫卡溪流:“;更新的群集元数据版本“;跑了很长时间,apache-kafka,apache-kafka-streams,Apache Kafka,Apache Kafka Streams,在过去的几个小时里,我的Kafka Streams作业试图加快速度,但却被一个非常大的主题的更新的集群元数据版本所占据: 2018-01-03T13:30:56.871 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer] - Starting the Kafka consumer 2018-01-03T13:30:56.882 DEBUG [org.apache.kafka.clients.Metadata] - Updated clu

在过去的几个小时里,我的Kafka Streams作业试图加快速度,但却被一个非常大的主题的
更新的集群元数据版本所占据:

2018-01-03T13:30:56.871 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer] - Starting the Kafka consumer
2018-01-03T13:30:56.882 DEBUG [org.apache.kafka.clients.Metadata] - Updated cluster metadata version 1 to Cluster(id = null, nodes = [node2:9092 (id: -2 rack: null), 
  node3:9092 (id: -3 rack: null), node1:9092 (id: -1 rack: null)], partitions = [])
...
2018-01-04T10:10:27.945 DEBUG [org.apache.kafka.clients.NetworkClient] - Sending metadata request (type=MetadataRequest, topics=<ALL>) to node node3:9092 (id: 3 rack: null)
2018-01-04T10:10:27.952 DEBUG [org.apache.kafka.clients.Metadata] - Updated cluster metadata version 704157 to Cluster(id = DmwzpJYRTEGrm8MfjcpXLw, nodes = [node3:9092 (id: 3 rack: null), 
  node1:9092 (id: 1 rack: null), node2:9092 (id: 2 rack: null)], partitions = [Partition(topic = raw-messages, partition = 9, leader = 3, replicas = [3], isr = [3]), 
  Partition(topic = raw-messages, partition = 8, leader = none, replicas = [2], isr = []), 
  Partition(topic = raw-messages, partition = 7, leader = 1, replicas = [1], isr = [1]), 
  Partition(topic = raw-messages, partition = 6, leader = 3, replicas = [3], isr = [3]), 
  Partition(topic = raw-messages, partition = 5, leader = none, replicas = [2], isr = []), 
  Partition(topic = raw-messages, partition = 4, leader = 1, replicas = [1], isr = [1]), 
  Partition(topic = raw-messages, partition = 3, leader = 3, replicas = [3], isr = [3]), 
  Partition(topic = raw-messages, partition = 2, leader = none, replicas = [2], isr = []), 
  Partition(topic = raw-messages, partition = 1, leader = 1, replicas = [1], isr = [1]), 
  Partition(topic = raw-messages, partition = 0, leader = 3, replicas = [3], isr = [3])])
2018-01-03T13:30:56.871调试[org.apache.kafka.clients.consumer.KafkaConsumer]-启动卡夫卡消费程序
2018-01-03T13:30:56.882调试[org.apache.kafka.clients.Metadata]-将群集元数据版本1更新为群集(id=null,nodes=[node2:9092(id:-2机架:null),
节点3:9092(id:-3机架:空),节点1:9092(id:-1机架:空)],分区=[])
...
2018-01-04T10:10:27.945调试[org.apache.kafka.clients.NetworkClient]-将元数据请求(类型=MetadataRequest,主题=)发送到节点3:9092(id:3机架:null)
2018-01-04T10:10:27.952调试[org.apache.kafka.clients.Metadata]-将群集元数据版本704157更新到群集(id=DmwzpJYRTEGrm8MfjcpXLw,nodes=[node3:9092(id:3机架:null),
node1:9092(id:1机架:空)、node2:9092(id:2机架:空)),分区=[分区(主题=原始消息,分区=9,前导=3,副本=[3],isr=[3]),
分区(主题=原始消息,分区=8,前导=无,副本=[2],isr=[]),
分区(主题=原始消息,分区=7,领导者=1,副本=[1],isr=[1]),
分区(主题=原始消息,分区=6,前导=3,副本=[3],isr=[3]),
分区(主题=原始消息,分区=5,前导=无,副本=[2],isr=[]),
分区(主题=原始消息,分区=4,前导=1,副本=[1],isr=[1]),
分区(主题=原始消息,分区=3,前导=3,副本=[3],isr=[3]),
分区(主题=原始消息,分区=2,前导=无,副本=[2],isr=[]),
分区(主题=原始消息,分区=1,领导者=1,副本=[1],isr=[1]),
分区(主题=原始消息,分区=0,领导者=3,副本=[3],isr=[3]))
这是一个带有新应用程序id的新Kafka Streams作业,应该从一开始就开始(offset=0)

有几个事件可能导致这种行为:

  • 我们从融合平台3.2.0迁移到融合平台4.0.0
  • 由于数据中心断电,卡夫卡群集严重崩溃

我的问题是,我应该让它运行吗?目前,作业位于
元数据版本704157
,我可以找出或估计版本总数吗?

876649
并计数…
1451669
-它会结束吗?这表示内部使用的使用者找不到分区的前导。。。您应该检查Kafka群集的运行状况。比较leader的分区8、5、2 report
none
——因此消费者在找到leader之前一直处于停滞状态。