Kubernetes Kubeadm连接失败，并在设置节点批注以启用卷控制器连接/分离时挂起

我在获取节点以加入我的k8s群集时遇到问题。它以前在集群中，但现在拒绝加入。除了离开和重新加入集群之外，服务器本身没有任何变化

• Kubernetes版本：

  1.14.9beta

  （为与Kubespowner兼容而从源代码处编译）
• 操作系统版本：

  Manjaro Openbox

  （所有系统都处于相同的补丁级别）

下面是加入尝试的日志：

Nov 18 14:56:06 dragon-den kubelet[17701]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-con>Nov 18 14:56:06 dragon-den kubelet[17701]: Flag --allow-privileged has been deprecated, will be removed in a future version
Nov 18 14:56:06 dragon-den kubelet[17701]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubel>Nov 18 14:56:07 dragon-den kubelet[17701]: I1118 14:56:07.856840   17701 server.go:418] Version: v1.14.9-beta.0.44+500f5aba80d712
Nov 18 14:56:07 dragon-den kubelet[17701]: I1118 14:56:07.857493   17701 plugins.go:103] No cloud provider specified.
Nov 18 14:56:07 dragon-den kubelet[17701]: W1118 14:56:07.857566   17701 server.go:557] standalone mode, no API client
Nov 18 14:56:08 dragon-den kubelet[17701]: W1118 14:56:08.026479   17701 server.go:475] No api server defined - no events will be sent to API server.
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.026530   17701 server.go:629] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.027259   17701 container_manager_linux.go:261] container manager verified user specified cgroup-root exists: []
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.027322   17701 container_manager_linux.go:266] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime>Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.027617   17701 container_manager_linux.go:286] Creating device plugin manager: true
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.027819   17701 state_mem.go:36] [cpumanager] initializing new in-memory state store
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.036358   17701 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.036403   17701 client.go:104] Start docker client with request timeout=2m0s
Nov 18 14:56:08 dragon-den kubelet[17701]: W1118 14:56:08.063851   17701 docker_service.go:561] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.063918   17701 docker_service.go:238] Hairpin mode set to "hairpin-veth"
Nov 18 14:56:08 dragon-den kubelet[17701]: W1118 14:56:08.064142   17701 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
Nov 18 14:56:08 dragon-den kubelet[17701]: W1118 14:56:08.068056   17701 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection cleanup.
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.070412   17701 docker_service.go:253] Docker cri networking managed by kubernetes.io/no-op
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.086972   17701 docker_service.go:258] Docker Info: &{ID:UZIS:BXME:6KSB:34PI:HKEZ:HRT6:I4XW:44VI:AR5M:Q4P7:EGFA:KRDD Containers:2 ContainersRunning:0 ContainersPaused:0 ContainersS>Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.087132   17701 docker_service.go:271] Setting cgroupDriver to systemd
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.099887   17701 remote_runtime.go:62] parsed scheme: ""
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.099950   17701 remote_runtime.go:62] scheme "" not registered, fallback to default scheme
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.100029   17701 remote_image.go:50] parsed scheme: ""
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.100049   17701 remote_image.go:50] scheme "" not registered, fallback to default scheme
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.100315   17701 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{/var/run/dockershim.sock 0  <nil>}]
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.100356   17701 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{/var/run/dockershim.sock 0  <nil>}]
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.100381   17701 clientconn.go:796] ClientConn switching balancer to "pick_first"
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.100416   17701 clientconn.go:796] ClientConn switching balancer to "pick_first"
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.100519   17701 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0xc0000cbaa0, CONNECTING
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.100575   17701 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0xc000349b20, CONNECTING
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.100959   17701 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0xc000349b20, READY
Nov 18 14:56:08 dragon-den kubelet[17701]: I1118 14:56:08.101021   17701 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0xc0000cbaa0, READY
Nov 18 14:56:28 dragon-den kubelet[17701]: E1118 14:56:28.454199   17701 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
Nov 18 14:56:28 dragon-den kubelet[17701]:         For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.488935   17701 kuberuntime_manager.go:210] Container runtime docker initialized, version: 19.03.4-ce, apiVersion: 1.40.0
Nov 18 14:56:28 dragon-den kubelet[17701]: W1118 14:56:28.489074   17701 volume_host.go:77] kubeClient is nil. Skip initialization of CSIDriverLister
Nov 18 14:56:28 dragon-den kubelet[17701]: W1118 14:56:28.489559   17701 csi_plugin.go:215] kubernetes.io/csi: kubeclient not set, assuming standalone kubelet
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.509315   17701 server.go:1055] Started kubelet
Nov 18 14:56:28 dragon-den kubelet[17701]: W1118 14:56:28.509368   17701 kubelet.go:1387] No api server defined - no node status update will be sent.
Nov 18 14:56:28 dragon-den kubelet[17701]: E1118 14:56:28.509369   17701 kubelet.go:1282] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory ca>Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.509441   17701 server.go:141] Starting to listen on 0.0.0.0:10250
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.510125   17701 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.510164   17701 status_manager.go:148] Kubernetes client is nil, not starting status manager.
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.510176   17701 kubelet.go:1808] Starting kubelet main sync loop.
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.510201   17701 volume_manager.go:248] Starting Kubelet Volume Manager
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.510197   17701 kubelet.go:1825] skipping pod synchronization - [container runtime status check may not have completed yet., PLEG is not healthy: pleg has yet to be successful.]    Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.510272   17701 desired_state_of_world_populator.go:130] Desired state populator starts to run
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.510546   17701 server.go:343] Adding debug handlers to kubelet server.
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.610328   17701 kubelet.go:1825] skipping pod synchronization - container runtime status check may not have completed yet.
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.667490   17701 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.708302   17701 cpu_manager.go:155] [cpumanager] starting with none policy
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.708327   17701 cpu_manager.go:156] [cpumanager] reconciling every 10s
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.708362   17701 policy_none.go:42] [cpumanager] none policy: Start
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.710493   17701 reconciler.go:154] Reconciler: start to sync state
Nov 18 14:56:28 dragon-den kubelet[17701]: W1118 14:56:28.732840   17701 container.go:422] Failed to get RecentStats("/libcontainer_17701_systemd_test_default.slice") while determining the next housekeeping: unable to find data in memory>Nov 18 14:56:28 dragon-den kubelet[17701]: W1118 14:56:28.783045   17701 manager.go:540] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
Nov 18 14:56:28 dragon-den kubelet[17701]: I1118 14:56:28.783441   17701 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach

服务器配置：

###
# kubernetes kubelet (minion) config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"

# The port for the info server to serve on
# KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
#KUBELET_HOSTNAME="--hostname-override=127.0.0.1"

# location of the api-server
#KUBELET_API_SERVER="--api-servers=http://127.0.0.1:8080"

# Add your own!
KUBELET_ARGS="--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf \
              --kubeconfig=/etc/kubernetes/kubelet.conf \
              --config=/var/lib/kubelet/config.yaml \
              --network-plugin=cni \
              --pod-infra-container-image=k8s.gcr.io/pause:3.1
              --cgroup-driver=systemd"

---

我也在使用印花棉布进行联网，如果这有什么不同的话，这是一个迹象，表明当这个节点是集群的一部分时，您还留下了一些东西。在大多数情况下，这不是强制性的，但我强烈建议在从集群中删除节点后，除了运行

kubeadm reset

之外，还运行一些命令来清理节点

sudo kubeadm reset -f
sudo systemctl stop kubelet
sudo systemctl stop docker
sudo rm -rf /var/lib/cni/
sudo rm -rf /var/lib/kubelet/*
sudo rm -rf /var/lib/etcd/*
sudo rm -rf /etc/cni/
sudo ifconfig cni0 down
sudo ifconfig flannel.1 down
sudo ifconfig docker0 down
sudo ip link delete cni0
sudo ip link delete flannel.1
sudo systemctl start docker

这些命令中的一些可能会失败，具体取决于您的实现

我发现这是为这个目的而创造的。它有点旧，但你可以从中吸取一些东西

---

这是一个与您类似的案例。

这是一个迹象，表明当此节点是集群的一部分时，您留下了一些东西。在大多数情况下，这不是强制性的，但我强烈建议在从集群中删除节点后，除了运行

kubeadm reset

之外，还运行一些命令来清理节点

sudo kubeadm reset -f
sudo systemctl stop kubelet
sudo systemctl stop docker
sudo rm -rf /var/lib/cni/
sudo rm -rf /var/lib/kubelet/*
sudo rm -rf /var/lib/etcd/*
sudo rm -rf /etc/cni/
sudo ifconfig cni0 down
sudo ifconfig flannel.1 down
sudo ifconfig docker0 down
sudo ip link delete cni0
sudo ip link delete flannel.1
sudo systemctl start docker

这些命令中的一些可能会失败，具体取决于您的实现

我发现这是为这个目的而创造的。它有点旧，但你可以从中吸取一些东西

---

这是一个与您类似的案例。

请描述一下您从集群中删除此节点并将其添加回集群的步骤好吗？@mWatney我使用

kubeadm reset

重置所有配置，以及清理iptable并验证

/var/lib/etcd

为空。重新加入它只是使用正常的

kubeadm join

命令。请运行以下命令并重试<代码>$sudo kubeadm重置-f；sudo systemctl stop kubelet；sudo-systemctl-stop-docker；sudorm-rf/var/lib/cni/；sudo rm-rf/var/lib/kubelet/*；sudom-rf/var/lib/etcd/*；sudo rm-rf/etc/cni/；sudo-ifconfig-cni0-down；sudo ifconfig法兰绒1向下；sudo-ifconfig-docker0关闭；sudo ip链路删除cni0；sudoip链接删除flannel.1；sudo systemctl start docker根据您的设置，有些可能会失败。我们的想法是清理一些遗留的东西。请描述一下您从集群中删除此节点并将其添加回集群的步骤好吗？@mWatney我使用

kubeadm reset

重置所有配置，以及清理iptable并验证

/var/lib/etcd

为空。重新加入它只是使用正常的

kubeadm join

命令。请运行以下命令并重试<代码>$sudo kubeadm重置-f；sudo systemctl stop kubelet；sudo-systemctl-stop-docker；sudorm-rf/var/lib/cni/；sudo rm-rf/var/lib/kubelet/*；sudom-rf/var/lib/etcd/*；sudo rm-rf/etc/cni/；sudo-ifconfig-cni0-down；sudo ifconfig法兰绒1向下；sudo-ifconfig-docker0关闭；sudo ip链路删除cni0；sudoip链接删除flannel.1；sudo systemctl start docker根据您的设置，有些可能会失败。我们的想法是清理更多的遗留物。

###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers
#KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://127.0.0.1:8080"

sudo kubeadm reset -f
sudo systemctl stop kubelet
sudo systemctl stop docker
sudo rm -rf /var/lib/cni/
sudo rm -rf /var/lib/kubelet/*
sudo rm -rf /var/lib/etcd/*
sudo rm -rf /etc/cni/
sudo ifconfig cni0 down
sudo ifconfig flannel.1 down
sudo ifconfig docker0 down
sudo ip link delete cni0
sudo ip link delete flannel.1
sudo systemctl start docker