Fatal编程技术网
  • kubernetes/
  • Kubernetes 在Kubernete1.15.4中，印花棉布吊舱准备就绪探针和活性探针始终失败

Kubernetes 在Kubernete1.15.4中，印花棉布吊舱准备就绪探针和活性探针始终失败

kubernetes

Kubernetes 在Kubernete1.15.4中，印花棉布吊舱准备就绪探针和活性探针始终失败,kubernetes,rancher,calico,Kubernetes,Rancher,Calico,在我将rancher从v2.2.8升级到v2.3之后，我遇到了这个问题。集群通过使用rke v0.3.0进行部署。我得到的第一个问题是准备就绪探测失败：HTTP探测失败，状态代码为：503下面的日志中有一个问题 但是在我遵循解决方案并添加解决方案中提到的crd.yml之后。我已经得到下面的错误日志 kube-system Warning Unhealthy canal-8grhd Readiness probe failed: Get http://localhost:9

在我将rancher从v2.2.8升级到v2.3之后，我遇到了这个问题。集群通过使用rke v0.3.0进行部署。我得到的第一个问题是
准备就绪探测失败：HTTP探测失败，状态代码为：503
下面的日志中有一个问题

但是在我遵循解决方案并添加解决方案中提到的
crd.yml
之后。我已经得到下面的错误日志

kube-system    Warning    Unhealthy    canal-8grhd    Readiness probe failed: Get http://localhost:9099/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)    a minute ago
kube-system    Warning    Unhealthy    canal-q2j9v    Liveness probe failed: Get http://localhost:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)    3 minutes ago
kube-system    Warning    Unhealthy    canal-tz42g    Readiness probe failed: Get http://localhost:9099/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)    3 minutes ago
kube-system    Warning    Unhealthy    canal-5svn7    Liveness probe failed: Get http://localhost:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)    3 minutes ago
kube-system    Warning    Unhealthy    canal-v7wmv    Readiness probe failed: Get http://localhost:9099/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)    4 minutes ago
kube-system    Warning    Unhealthy    canal-8grhd    Liveness probe failed: Get http://localhost:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)    6 minutes ago
kube-system    Warning    Unhealthy    canal-q2j9v    Readiness probe failed: Get http://localhost:9099/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)    8 minutes ago
kube-system    Warning    Unhealthy    canal-tz42g    Liveness probe failed: Get http://localhost:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)    8 minutes ago
kube-system    Warning    Unhealthy    canal-v7wmv    Liveness probe failed: Get http://localhost:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)    9 minutes ago
kube-system    Warning    Unhealthy    canal-5svn7    Readiness probe failed: Get http://localhost:9099/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)    13 minutes ago
运河吊舱的yaml

Name:           canal-q2j9v
Namespace:      kube-system
Priority:       0
Node:           woker2/10.16.18.25
Start Time:     Wed, 16 Oct 2019 19:05:02 +0800
Labels:         controller-revision-hash=cd754f475
                k8s-app=canal
                pod-template-generation=2
Annotations:    scheduler.alpha.kubernetes.io/critical-pod:
Status:         Running
IP:             10.16.18.25
Controlled By:  DaemonSet/canal
Init Containers:
  install-cni:
    Container ID:  docker://262db0783d8b140e45d47faf8bdf2d1d6bd3f2d858d2c9d7985e16bf1a8f0f4d
    Image:         rancher/calico-cni:v3.7.4
    Image ID:      docker-pullable://rancher/calico-cni@sha256:5dc320eece42a8a1184bc5633e8779dcdd06b8a3ac010eefc93a9e38859b235a
    Port:          <none>
    Host Port:     <none>
    Command:
      /install-cni.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 16 Oct 2019 19:05:03 +0800
      Finished:     Wed, 16 Oct 2019 19:05:04 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      CNI_CONF_NAME:         10-canal.conflist
      CNI_NETWORK_CONFIG:    <set to the key 'cni_network_config' of config map 'canal-config'>  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from canal-token-pzx9p (ro)
Containers:
  calico-node:
    Container ID:   docker://df955a12ef388c86828cbc6de7b9587f45bf3654578ad21789fbb1c16f38db8f
    Image:          rancher/calico-node:v3.7.4
    Image ID:       docker-pullable://rancher/calico-node@sha256:709c559e53021355a19efdb57981bebddd96f35628dc8b49a5f9af8561d8497c
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 16 Oct 2019 19:05:04 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      250m
    Liveness:   http-get http://localhost:9099/liveness delay=10s timeout=1s period=10s #success=1 #failure=6
    Readiness:  http-get http://localhost:9099/readiness delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          none
      CLUSTER_TYPE:                       k8s,canal
      FELIX_IPTABLESREFRESHINTERVAL:      60
      IP:
      CALICO_IPV4POOL_CIDR:               192.168.0.0/16
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_LOGFILEPATH:                  none
      FELIX_LOGSEVERITYSYS:
      FELIX_LOGSEVERITYSCREEN:            Warning
      FELIX_HEALTHENABLED:                true
    Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from canal-token-pzx9p (ro)
  kube-flannel:
    Container ID:  docker://97d6b9a9e8e5736535182559660cd3c291fb748a0dd3f121eb98afa86817d622
    Image:         rancher/coreos-flannel:v0.11.0
    Image ID:      docker-pullable://rancher/coreos-flannel@sha256:bd76b84c74ad70368a2341c2402841b75950df881388e43fc2aca000c546653a
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/bin/flanneld
      --ip-masq
      --kube-subnet-mgr
    State:          Running
      Started:      Wed, 16 Oct 2019 19:05:05 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:          canal-q2j9v (v1:metadata.name)
      POD_NAMESPACE:     kube-system (v1:metadata.namespace)
      FLANNELD_IFACE:    <set to the key 'canal_iface' of config map 'canal-config'>  Optional: false
      FLANNELD_IP_MASQ:  <set to the key 'masquerade' of config map 'canal-config'>   Optional: false
    Mounts:
      /etc/kube-flannel/ from flannel-cfg (rw)
      /run/xtables.lock from xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from canal-token-pzx9p (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  flannel-cfg:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      canal-config
    Optional:  false
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
  canal-token-pzx9p:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  canal-token-pzx9p
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     :NoSchedule
                 :NoExecute
                 CriticalAddonsOnly
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason     Age                    From             Message
  ----     ------     ----                   ----             -------
  Warning  Unhealthy  3m1s (x243 over 19h)   kubelet, woker2  Liveness probe failed: Get http://localhost:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  2m58s (x419 over 21h)  kubelet, woker2  Readiness probe failed: Get http://localhost:9099/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

名称：canal-q2j9v
名称空间：kube系统
优先级：0
节点：woker2/10.16.18.25
开始时间：2019年10月16日星期三19:05:02+0800
标签：控制器修订哈希=cd754f475
k8s app=运河
pod模板生成=2
注释：scheduler.alpha.kubernetes.io/critical-pod:
状态：正在运行
IP:10.16.18.25
控制者：守护程序/通道
初始化容器：
安装cni：
容器ID:docker://262db0783d8b140e45d47faf8bdf2d1d6bd3f2d858d2c9d7985e16bf1a8f0f4d
图片：rancher/calico cni:v3.7.4
图像ID:docker-pullable://rancher/calico-cni@sha256:5dc320eece42a8a1184bc5633e8779dcdd06b8a3ac010eefc93a9e38859b235a
端口：
主机端口：
命令：
/安装-cni.sh
国家：终止
原因：已完成
退出代码：0
开始时间：2019年10月16日星期三19:05:03+0800
完成日期：2019年10月16日星期三19:05:04+0800
准备好了吗
重新启动计数：0
环境：
CNI_形态名称：10-canal.conflist
CNI_网络_配置：可选：false
KUBERNETES_节点_名称：（v1:spec.nodeName）
睡眠：错误
挂载：
/主机/etc/cni/net.d来自cni网络目录（rw）
/主机/opt/cni/bin来自cni-bin-dir（rw）
/var/run/secrets/kubernetes.io/serviceCount来自canal-token-pzx9p（ro）
容器：
印花布节点：
容器ID:docker://df955a12ef388c86828cbc6de7b9587f45bf3654578ad21789fbb1c16f38db8f
图：rancher/calico节点：v3.7.4
图像ID:docker-pullable://rancher/calico-node@sha256:709C559E53021355A19EFDB57981BEBDD96F35628DC8B49A5F9AF8561D8497C
端口：
主机端口：
状态：正在运行
开始时间：2019年10月16日星期三19:05:04+0800
准备好了吗
重新启动计数：0
请求：
中央处理器：250米
活跃度：http gethttp://localhost:9099/liveness 延迟=10s超时=1s周期=10s#成功=1#失败=6
准备就绪：http gethttp://localhost:9099/readiness 延迟=0s超时=1s周期=10s成功=1失败=3
环境：
数据存储类型：kubernetes
等待数据存储：true
节点名称：（v1:spec.NODENAME）
印花布网络后端：无
集群类型：k8s，运河
FELIX_iptables刷新间隔：60
知识产权：
印花布IPV4池CIDR:192.168.0.0/16
印花布禁用文件日志记录：true
FELIX_DEFAULTENDPOINTTOHOSTACTION:接受
FELIX_IPv6支持：错误
FELIX_LOGFILEPATH:无
FELIX_LOGSEVERITYSYS：
FELIX_LogSeverity屏幕：警告
FELIX_HEALTHENABLED：正确
挂载：
/库/来自库模块（ro）的模块
/从xtables锁（rw）运行/xtables.lock
/var/lib/calico来自var-lib calico（rw）
/var/run/calico来自var run calico（rw）
/var/run/secrets/kubernetes.io/serviceCount来自canal-token-pzx9p（ro）
库贝法兰绒：
容器ID:docker://97d6b9a9e8e5736535182559660cd3c291fb748a0dd3f121eb98afa86817d622
图片：rancher/coreos法兰绒：v0.11.0
图像ID:docker-pullable://rancher/coreos-flannel@sha256:bd76b84c74ad70368a2341c2402841b75950df881388e43fc2aca000c546653a
端口：
主机端口：
命令：
/可选/箱/法兰绒
--ip masq
--kube子网经理
状态：正在运行
开始时间：2019年10月16日星期三19:05:05+0800
准备好了吗
重新启动计数：0
环境：
POD_名称：canal-q2j9v（v1:metadata.NAME）
POD_名称空间：kube系统（v1:metadata.NAMESPACE）
法兰布：可选：false
FLANNELD_IP_MASQ:可选：false
挂载：
/etc/kube法兰绒/来自法兰绒cfg（rw）
/从xtables锁（rw）运行/xtables.lock
/var/run/secrets/kubernetes.io/serviceCount来自canal-token-pzx9p（ro）
条件：
类型状态
初始化为True
准备好了吗
集装箱准备好了吗
播客预定为真
卷数：
lib模块：
类型：主机路径（裸主机目录卷）
路径：/lib/模块
主机路径类型：
var run印花布：
类型：主机路径（裸主机目录卷）
路径：/var/run/calico
主机路径类型：
var lib印花布：
类型：主机路径（裸主机目录卷）
路径：/var/lib/calico
主机路径类型：
xtables锁：
类型：主机路径（裸主机目录卷）
路径：/run/xtables.lock
主机路径类型：文件或创建
法兰绒：
类型：ConfigMap（由ConfigMap填充的卷）
名称：canal config
可选：false
cni bin dir：
类型：主机路径（裸主机目录卷）
路径：/opt/cni/bin
主机路径类型：
网络总监：
类型：主机路径（裸主机目录卷）
路径：/etc/cni/net.d
主机路径类型：
canal-token-pzx9p：
类型：Secret（由Secret填充的卷）
秘书名：canal-token-pzx9p
可选：false
QoS等级：Burstable
节点选择器：
容忍：：NoSchedule
：不执行
批判地
node.kubernetes.io/磁盘压力：NoSchedule
node.kubernetes.io/memory-pre

Name:           canal-q2j9v
Namespace:      kube-system
Priority:       0
Node:           woker2/10.16.18.25
Start Time:     Wed, 16 Oct 2019 19:05:02 +0800
Labels:         controller-revision-hash=cd754f475
                k8s-app=canal
                pod-template-generation=2
Annotations:    scheduler.alpha.kubernetes.io/critical-pod:
Status:         Running
IP:             10.16.18.25
Controlled By:  DaemonSet/canal
Init Containers:
  install-cni:
    Container ID:  docker://262db0783d8b140e45d47faf8bdf2d1d6bd3f2d858d2c9d7985e16bf1a8f0f4d
    Image:         rancher/calico-cni:v3.7.4
    Image ID:      docker-pullable://rancher/calico-cni@sha256:5dc320eece42a8a1184bc5633e8779dcdd06b8a3ac010eefc93a9e38859b235a
    Port:          <none>
    Host Port:     <none>
    Command:
      /install-cni.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 16 Oct 2019 19:05:03 +0800
      Finished:     Wed, 16 Oct 2019 19:05:04 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      CNI_CONF_NAME:         10-canal.conflist
      CNI_NETWORK_CONFIG:    <set to the key 'cni_network_config' of config map 'canal-config'>  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from canal-token-pzx9p (ro)
Containers:
  calico-node:
    Container ID:   docker://df955a12ef388c86828cbc6de7b9587f45bf3654578ad21789fbb1c16f38db8f
    Image:          rancher/calico-node:v3.7.4
    Image ID:       docker-pullable://rancher/calico-node@sha256:709c559e53021355a19efdb57981bebddd96f35628dc8b49a5f9af8561d8497c
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 16 Oct 2019 19:05:04 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      250m
    Liveness:   http-get http://localhost:9099/liveness delay=10s timeout=1s period=10s #success=1 #failure=6
    Readiness:  http-get http://localhost:9099/readiness delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          none
      CLUSTER_TYPE:                       k8s,canal
      FELIX_IPTABLESREFRESHINTERVAL:      60
      IP:
      CALICO_IPV4POOL_CIDR:               192.168.0.0/16
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_LOGFILEPATH:                  none
      FELIX_LOGSEVERITYSYS:
      FELIX_LOGSEVERITYSCREEN:            Warning
      FELIX_HEALTHENABLED:                true
    Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from canal-token-pzx9p (ro)
  kube-flannel:
    Container ID:  docker://97d6b9a9e8e5736535182559660cd3c291fb748a0dd3f121eb98afa86817d622
    Image:         rancher/coreos-flannel:v0.11.0
    Image ID:      docker-pullable://rancher/coreos-flannel@sha256:bd76b84c74ad70368a2341c2402841b75950df881388e43fc2aca000c546653a
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/bin/flanneld
      --ip-masq
      --kube-subnet-mgr
    State:          Running
      Started:      Wed, 16 Oct 2019 19:05:05 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:          canal-q2j9v (v1:metadata.name)
      POD_NAMESPACE:     kube-system (v1:metadata.namespace)
      FLANNELD_IFACE:    <set to the key 'canal_iface' of config map 'canal-config'>  Optional: false
      FLANNELD_IP_MASQ:  <set to the key 'masquerade' of config map 'canal-config'>   Optional: false
    Mounts:
      /etc/kube-flannel/ from flannel-cfg (rw)
      /run/xtables.lock from xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from canal-token-pzx9p (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  flannel-cfg:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      canal-config
    Optional:  false
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
  canal-token-pzx9p:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  canal-token-pzx9p
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     :NoSchedule
                 :NoExecute
                 CriticalAddonsOnly
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason     Age                    From             Message
  ----     ------     ----                   ----             -------
  Warning  Unhealthy  3m1s (x243 over 19h)   kubelet, woker2  Liveness probe failed: Get http://localhost:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  2m58s (x419 over 21h)  kubelet, woker2  Readiness probe failed: Get http://localhost:9099/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)