Docker 卡夫卡无法与zookeeper连接,出现错误“;状态为“正在连接”时等待连接超时;
我正试着用库伯内特斯豆荚管理我的卡夫卡和动物园管理员 这是我的动物园管理员服务。yaml:Docker 卡夫卡无法与zookeeper连接,出现错误“;状态为“正在连接”时等待连接超时;,docker,kubernetes,apache-kafka,apache-zookeeper,google-kubernetes-engine,Docker,Kubernetes,Apache Kafka,Apache Zookeeper,Google Kubernetes Engine,我正试着用库伯内特斯豆荚管理我的卡夫卡和动物园管理员 这是我的动物园管理员服务。yaml: apiVersion: v1 kind: Service metadata: annotations: kompose.cmd: kompose convert kompose.version: 1.1.0 (36652f6) creationTimestamp: null labels: io.kompose.service: zookeeper-svc name
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.1.0 (36652f6)
creationTimestamp: null
labels:
io.kompose.service: zookeeper-svc
name: zookeeper-svc
spec:
ports:
- name: "2181"
port: 2181
targetPort: 2181
selector:
io.kompose.service: zookeeper
status:
loadBalancer: {}
下面是zookeeper部署。yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.1.0 (36652f6)
creationTimestamp: null
labels:
io.kompose.service: zookeeper
name: zookeeper
spec:
replicas: 1
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
io.kompose.service: zookeeper
spec:
containers:
- image: wurstmeister/zookeeper
name: zookeeper
ports:
- containerPort: 2181
resources: {}
restartPolicy: Always
status: {}
kafka部署。yaml
如下:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert -f docker-compose.yml
kompose.version: 1.1.0 (36652f6)
creationTimestamp: null
labels:
io.kompose.service: kafka
name: kafka
spec:
replicas: 1
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
io.kompose.service: kafka
spec:
containers:
- env:
- name: KAFKA_ADVERTISED_HOST_NAME
value: kafka
- name: KAFKA_ZOOKEEPER_CONNECT
value: zookeeper:2181
- name: KAFKA_PORT
value: "9092"
- name: KAFKA_ZOOKEEPER_CONNECT_TIMEOUT_MS
value: "60000"
image: wurstmeister/kafka
name: kafka
ports:
- containerPort: 9092
resources: {}
restartPolicy: Always
status: {}
我首先启动zookeeper服务和部署。一旦zookeeper启动并且kubectl get pods
显示它处于运行状态,我就启动kafka部署。由于restartPolicy一如既往,Kafka部署开始失败并一次又一次地重新启动。当我检查kafka docker的日志时,我发现它无法连接到zookeeper服务,并且连接超时。这是卡夫卡容器中的日志
[2018-09-03 07:06:06,670] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
atkafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$ waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:230)
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:226)
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:226)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:226)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:95)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1580)
at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:348)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:75)
at kafka.Kafka.main(Kafka.scala)
[2018-09-03 07:06:06,671] INFO shutting down (kafka.server.KafkaServer)
[2018-09-03 07:06:06,673] WARN (kafka.utils.CoreUtils$)
java.lang.NullPointerException
atkafka.server.KafkaServer$$anonfun$shutdown$5.apply$mcV$sp(KafkaServer.scala:579)
at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:86)
at kafka.server.KafkaServer.shutdown(KafkaServer.scala:579)
at kafka.server.KafkaServer.startup(KafkaServer.scala:329)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:75)
at kafka.Kafka.main(Kafka.scala)
[2018-09-03 07:06:06,676] INFO shut down completed
(kafka.server.KafkaServer)
[2018-09-03 07:06:06,677] ERROR Exiting Kafka.
(kafka.server.KafkaServerStartable)
[2018-09-03 07:06:06,678] INFO shutting down
(kafka.server.KafkaServer)
kubectl get svc-o wide的输出如下所示:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 50m <none>
zookeeper ClusterIP 10.98.180.138 <none> 2181/TCP 48m io.kompose.service=zookeeper
编辑:
kubectl描述卡夫卡-757dc6c47b-zpzfz吊舱的输出
Name: kafka-757dc6c47b-zpzfz
Namespace: default
Node: administrator-thinkpad-l480/10.11.17.86
Start Time: Wed, 05 Sep 2018 16:17:06 +0530
Labels: io.kompose.service=kafka
pod-template-hash=3138727036
Annotations: <none>
Status: Running
IP: 10.32.0.17
Controlled By: ReplicaSet/kafka-757dc6c47b
Containers:
kafka:
Container ID:docker://2bdc06d876ae23437c61f4e95539a67903cdb61e88fd9c68377b47c7705293a3
Image: wurstmeister/kafka
Image ID: docker-pullable://wurstmeister/kafka@sha256:2e3ff64e70ea983530f590282f36991c0a1b105350510f53cc3d1a0279b83c28
Port: 9092/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 05 Sep 2018 17:29:06 +0530
Finished: Wed, 05 Sep 2018 17:29:14 +0530
Ready: False
Restart Count: 18
Environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT_TIMEOUT_MS: 160000
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-nhb9z (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-nhb9z:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-nhb9z
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 3m (x293 over 1h) kubelet, administrator-thinkpad-l480 Back-off restarting failed container
名称:卡夫卡-757dc6c47b-zpzfz
名称空间:默认值
节点:administrator-thinkpad-l480/10.11.17.86
开始时间:2018年9月5日星期三16:17:06+0530
标签:io.kompose.service=kafka
pod模板哈希=3138727036
注释:
状态:正在运行
IP:10.32.0.17
控制人:ReplicaSet/kafka-757dc6c47b
容器:
卡夫卡:
容器ID:docker://2bdc06d876ae23437c61f4e95539a67903cdb61e88fd9c68377b47c7705293a3
图片:wurstmeister/kafka
图像ID:docker-pullable://wurstmeister/kafka@sha256:2E3FF64E70EA983530F590282F36991C0A1B105350510F53C3D1A0279B83C28
端口:9092/TCP
国家:等待
原因:仓促退却
最后状态:终止
原因:错误
退出代码:1
开始时间:2018年9月5日星期三17:29:06+0530
完成时间:2018年9月5日星期三17:29:14+0530
就绪:错误
重新启动计数:18
环境:
卡夫卡主持人姓名:卡夫卡
卡夫卡大学动物园管理员连接:动物园管理员:2181
卡夫卡港:9092
卡夫卡·动物园管理员·连接·超时·毫秒:160000
挂载:
/来自default-token-nhb9z(ro)的var/run/secrets/kubernetes.io/serviceCount
条件:
类型状态
初始化为True
准备错误
集装箱准备好了吗
播客预定为真
卷数:
default-token-nhb9z:
类型:Secret(由Secret填充的卷)
SecretName:default-token-nhb9z
可选:false
QoS等级:最佳努力
节点选择器:
容差:node.kubernetes.io/未就绪:不执行300秒
node.kubernetes.io/不可访问:不执行300秒
活动:
从消息中键入原因年龄
---- ------ ---- ---- -------
警告后退3m(x293超过1小时)kubelet,管理员-thinkpad-l480后退重新启动失败的容器
这可能是什么原因?以及解决方案
原因隐藏在以下日志行后面:
INFO Connecting to zookeeper on zookeeper:2181 (kafka.server.KafkaServer)
卡夫卡正在搜索zookeeper
,而它应该搜索您的服务名称,即zookeeper svc
解决方案很简单:在zookeeper服务中将zookeeper服务从zookeeper svc
重命名为zookeeper
。yaml
如下:
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.1.0 (36652f6)
creationTimestamp: null
labels:
io.kompose.service: zookeeper
name: zookeeper
spec:
ports:
- name: "2181"
port: 2181
targetPort: 2181
selector:
io.kompose.service: zookeeper
status:
loadBalancer: {}
或者,更简洁地说,保持服务名称不变,并重新配置Kafka以搜索zookeeper svc
,而不是zookeeper
注意:由于您的最小示例足以重现问题(谢谢,很高兴看到最小示例工作!),尽管pod已启动并正在运行(未处于错误状态),但它仍然显示:
java.io.IOException:无法解析与另一个问题相关的地址:kafka:9092
(服务不包括kafka部署)这超出了这个问题的范围,只是想让你知道 我在使用microk8s时发出了这个警告
警告:IPtables转发策略已删除
考虑通过以下方式启用流量转发:
sudo iptables-p向前接受
在我的情况下,我试图在启动Zookeeper之前启动Kafka服务器
所以正确的顺序是先启动Zookeeper,然后启动Kafka服务器。Zookeeper日志报告了什么?用Zookeeper日志更新了问题Zookeeper很好。。。你试过使用合流头盔图表吗?@cricket\u 007不,我没有使用头盔图表……因为我从来没有听说过它(我也是库伯内特斯的新手)。但一个简短的概述暗示它有自己的卡夫卡形象。。反正我也不想用。还有其他方法吗?Confluent是卡夫卡的企业公司和主要开发人员,所以我不确定你对他们的卡夫卡图像有什么不满我在将
zookeeper svc
更改为zookeeper
后尝试过,但仍然面临同样的错误。奇怪。我已经运行了您的确切清单,docker edge上只有重命名的服务(来自answer),它正确启动并运行了2h+(注释中的错误从日志中列出)。您确定重新创建的服务清单正确吗?你能确保你有正确的清单和正确的创建顺序吗?你们能让助手容器来验证zookeeper:2181是否可以访问、启动并运行吗?事实上,为了正确起见:1)我已经运行了你们的确切清单来重现错误。2) 我删除了zookeeper服务,然后3)对其进行编辑,在两个位置用zookeeper替换zookeeper svc,4)重新创建了服务,5)删除并创建了kafka部署。在那些确切的步骤之后,所有的一切仍然在运行……我遵循了同样的步骤,仍然面临着同样的问题。还有,让助手容器验证zookeeper:2181是可访问的是什么意思?在同一命名空间中启动任何其他容器(busybox或其他任何容器),然后执行它以验证您可以访问它
Name: kafka-757dc6c47b-zpzfz
Namespace: default
Node: administrator-thinkpad-l480/10.11.17.86
Start Time: Wed, 05 Sep 2018 16:17:06 +0530
Labels: io.kompose.service=kafka
pod-template-hash=3138727036
Annotations: <none>
Status: Running
IP: 10.32.0.17
Controlled By: ReplicaSet/kafka-757dc6c47b
Containers:
kafka:
Container ID:docker://2bdc06d876ae23437c61f4e95539a67903cdb61e88fd9c68377b47c7705293a3
Image: wurstmeister/kafka
Image ID: docker-pullable://wurstmeister/kafka@sha256:2e3ff64e70ea983530f590282f36991c0a1b105350510f53cc3d1a0279b83c28
Port: 9092/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 05 Sep 2018 17:29:06 +0530
Finished: Wed, 05 Sep 2018 17:29:14 +0530
Ready: False
Restart Count: 18
Environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT_TIMEOUT_MS: 160000
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-nhb9z (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-nhb9z:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-nhb9z
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 3m (x293 over 1h) kubelet, administrator-thinkpad-l480 Back-off restarting failed container
INFO Connecting to zookeeper on zookeeper:2181 (kafka.server.KafkaServer)
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.1.0 (36652f6)
creationTimestamp: null
labels:
io.kompose.service: zookeeper
name: zookeeper
spec:
ports:
- name: "2181"
port: 2181
targetPort: 2181
selector:
io.kompose.service: zookeeper
status:
loadBalancer: {}