Kubernetes devk8s主机显示额外负载,导致无法获取POD的输出
我的开发人员k8s主机显示额外负载,导致无法获取POD的输出: admin@ip-172-20-49-150:~$kubectl获得订单-n cog阶段Kubernetes devk8s主机显示额外负载,导致无法获取POD的输出,kubernetes,kubectl,kubelet,Kubernetes,Kubectl,Kubelet,我的开发人员k8s主机显示额外负载,导致无法获取POD的输出: admin@ip-172-20-49-150:~$kubectl获得订单-n cog阶段 ^C admin@ip-172-20-49-150:~$ admin@ip-172-20-49-150:~$top top - 04:36:52 up 2 min, 2 users, load average: 14.39, 4.43, 1.55 Tasks: 140 total, 2 running, 138 sleeping,
^C
admin@ip-172-20-49-150:~$
admin@ip-172-20-49-150:~$top
top - 04:36:52 up 2 min, 2 users, load average: 14.39, 4.43, 1.55
Tasks: 140 total, 2 running, 138 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.2 sy, 0.0 ni, 0.0 id, 99.6 wa, 0.0 hi, 0.0 si, 0.2 st
KiB Mem: 3857324 total, 3778024 used, 79300 free, 192 buffers
KiB Swap: 0 total, 0 used, 0 free. 15680 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32 root 20 0 0 0 0 S 2.4 0.0 0:03.75 kswapd0
1263 root 20 0 97388 19036 0 S 1.3 0.5 0:01.06 kube-controller
1224 root 20 0 28764 11380 0 S 0.7 0.3 0:01.86 etcd
1358 root 20 0 46192 10608 0 S 0.7 0.3 0:00.69 kube-scheduler
1243 root 20 0 372552 343024 0 S 0.6 8.9 0:10.51 etcd
695 root 20 0 889180 52352 0 S 0.4 1.4 0:05.34 dockerd
752 root 20 0 205800 13756 0 S 0.4 0.4 0:00.56 protokube
816 root 20 0 449964 30804 0 S 0.4 0.8 0:02.26 kubelet
1247 root 20 0 3207664 2.856g 0 S 0.4 77.6 0:55.90 kube-apiserver
1279 root 20 0 40848 8900 0 S 0.4 0.2 0:00.46 kube-proxy
1 root 20 0 28788 1940 0 R 0.2 0.1 0:02.06 systemd
157 root 0 -20 0 0 0 S 0.2 0.0 0:00.06 kworker/1:1H
1562 admin 20 0 78320 1092 0 S 0.2 0.0 0:00.04 sshd
1585 admin 20 0 23660 540 0 R 0.2 0.0 0:00.11 top
1758 admin 20 0 33512 320 32 D 0.2 0.0 0:00.04 kubectl
1779 root 20 0 39368 436 0 D 0.2 0.0 0:00.01 docker-containe
请让我知道如何解决此问题
在主机上更新kubelet日志:
admin@ip-172-20-49-150:~$journalctl-u kubelet-f
Jan 06 05:41:44 ip-172-20-49-150 kubelet[819]: E0106 05:41:44.454586 819 pod_workers.go:182] Error syncing pod 685c903f9066f69a2e17c802cb043ed6 ("etcd-server-events-ip-172-20-49-150.us-west-1.compute.internal_kube-system(685c903f9066f69a2e17c802cb043ed6)"), skipping: failed to "StartContainer" for "etcd-container" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=etcd-container pod=etcd-server-events-ip-172-20-XX-XXX.us-west-1.compute.internal_kube-system(685c903f906b043ed6)"
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]: I0106 05:41:45.454266 819 kuberuntime_manager.go:500] Container {Name:kube-controller-manager Image:gcr.io/google_containers/kube-controller-manager:v1.8.4 Command:[/bin/sh -c /usr/local/bin/kube-controller-manager --allocate-node-cidrs=true --attach-detach-reconcile-sync-period=1m0s --cloud-provider=aws --cluster-cidr=100.96.0.0/11 --cluster-name=uw1b.k8s.ops.goldenratstud.io --cluster-signing-cert-file=/srv/kubernetes/ca.crt --cluster-signing-key-file=/srv/kubernetes/ca.key --configure-cloud-routes=true --kubeconfig=/var/lib/kube-controller-manager/kubeconfig --leader-elect=true --root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key --use-service-account-credentials=true --v=2 2>&1 | /bin/tee -a /var/log/kube-controller-manager.log] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI}]} VolumeMounts:[{Name:etcssl ReadOnly:true MountPath:/etc/ssl SubPath: MountPropagation:<nil>} {Name:etcpkitls ReadOnly:true MountPath:/etc/pki/tls SubPath: MountPropagation:<nil>} {Name:etcpkica-trust ReadOnly:true MountPath:/etc/pki/ca-trust SubPath: MountPropagation:<nil>} {Name:usrsharessl ReadOnly:true MountPath:/usr/share/ssl SubPath: MountPropagation:<nil>} {Name:usrssl ReadOnly:true MountPath:/usr/ssl SubPath: MountPropagation:<nil>} {Name:usrlibssl ReadOnly:true MountPath:/usr/lib/ssl SubPath: MountPropagation:<nil>} {Name:usrlocalopenssl ReadOnly:true MountPath:/usr/local/openssl SubPath: MountPropagation:<nil>} {Name:varssl ReadOnly:true MountPath:/var/ssl SubPath: MountPropagation:<nil>} {Name:etcopenssl ReadOnly:true MountPath:/etc/openssl SubPath: MountPropagation:<nil>} {Name:srvkube ReadOnly:true MountPath:/srv/kubernetes SubPath: MountPropagation:<nil>} {Name:logfile ReadOnly:false MountPath:/var/log/kube-controller-manager.log SubPath: MountPropagation:<nil>} {Name:varlibkcm ReadOnly:true MountPath:/var/lib/kube-controller-manager SubPath: MountPropagation:<nil>}] Live
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]: nessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:10252,Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]: I0106 05:41:45.454658 819 kuberuntime_manager.go:739] checking backoff for container "kube-controller-manager" in pod "kube-controller-manager-ip-172-20-49-150.us-west-1.compute.internal_kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)"
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]: I0106 05:41:45.454781 819 kuberuntime_manager.go:749] Back-off 5m0s restarting failed container=kube-controller-manager pod=kube-controller-manager-ip-172-20-49-150.us-west-1.compute.internal_kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]: E0106 05:41:45.454813 819 pod_workers.go:182] Error syncing pod ef6f03ef0b14d853dd38e4c2a5f426dc ("kube-controller-manager-ip-172-20-49-150.us-west-1.compute.internal_kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)"), skipping: failed to "StartContainer" for "kube-controller-manager" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kube-controller-manager pod=kube-controller-manager-ip-172-20-49-150.us-west-1.compute.internal_kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)"
Jan 06 05:41:47 ip-172-20-49-150 kubelet[819]: I0106 05:41:47.432074 819 container.go:471] Failed to update stats for container "/kubepods/burstable/pod2a5faee9437283d8ac7f396d86d07a03/0f62ea06693a7d4aaf6702d8ca370f2d5d2f1f3c4fdeab09aede15ea5311e47c": unable to determine device info for dir: /var/lib/docker/overlay/ce30183e915076727e708ed10b2ada4d55d1fe6d5c989c1cffc3e29cc00dad94: stat failed on /var/lib/docker/overlay/ce30183e915076727e708ed10b2ada4d55d1fe6d5c989c1cffc3e29cc00dad94 with error: no such file or directory, continuing to push stats
Jan 06 05:41:44 ip-172-20-49-150 kubelet[819]:E0106 05:41:44.454586 819 pod_-U-workers.go:182]错误同步pod 685c903f9066f69a2e17c802cb043ed6(“etcd-server-events-ip-172-20-49-150.us-west-1.compute.internal__-kube-system(685C903F9066F69A27C802CB043ED6)”),跳过:无法为“etcd容器”启动崩溃:“后退5ms重新启动失败的容器=etcd容器pod=etcd-server-events-ip-172-20-XX-XXX.us-west-1.compute.internal_kube-system(685c903f906b043ed6)”
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]:I0106 05:41:45.454266 819 kuberuntime_manager.go:500]容器{名称:kube控制器管理器图像:gcr.io/google_容器/kube控制器管理器:v1.8.4命令:[/bin/sh-c/usr/local/bin/kube controller manager--allocate node cidrs=true--attach detach-reconcil-sync period=1mos--cloud provider=aws--cluster cidr=100.96.0.0/11--cluster name=uw1b.k8s.ops.goldenratstud.io--cluster signing cert文件=/srv/kubernetes/ca.crt--cluster签名密钥文件=/srv/kubernetes/ca.key--configure cloud routes=--kubeconfig=/var/lib/kube controller manager/kubeconfig--leader elect=true--root ca file=/srv/kubernetes/ca.crt--service account private key file=/srv/kubernetes/server.key--use service account credentials=true--v=2>&1 |/bin/tee-a/var/log/kube controller manager.log]参数:[]工作目录:端口:[]环境:[]资源:[限制:映射[]请求:map[cpu:{i:{value:100 scale:-3}d:{Dec:}s:100m格式:小数点SI}]VolumeMounts:[{Name:etcssl ReadOnly:true MountPath:/etc/ssl子路径:MountPropagation:}{Name:etcpkitls ReadOnly:true MountPath:/etc/pki/tls子路径:mountpagation:}{Name:etcpkica trust ReadOnly:true MountPath:/etc/pki/ca trust子路径:mountpagation:}{Name:usrsharessl ReadOnly:true MountPath:/usr/share/ssl子路径:mountprogation:}{Name:usrssl ReadOnly:true MountPath:/usr/ssl子路径:mountprogation:}{Name:usrlibssl ReadOnly:true mountprogation:}{Name:usrlocalopenssl ReadOnly:true mountprogation:/usr/local/openssl子路径:mountprogation:}{Name:varssl ReadOnly:true MountPath:/var/ssl子路径:MountPropagation:}{Name:etcopenssl ReadOnly:true MountPath:/etc/openssl子路径:MountPropagation:}{Name:srvkube ReadOnly:true MountPath:/srv/kubernetes子路径:mountpagation:}{Name:logfile ReadOnly:false MountPath:/var/log/kube-controller-manager.log子路径:mountpagation:}{Name:varlibkcm ReadOnly:true MountPath:/var/lib/kube控制器管理器子路径:mountprogation:}]Live
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]:nessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,端口:10252,主机:127.0.0.1,方案:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodsSeconds:10,SuccessThreshold:1,FailureReshold:3,}ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}已死亡,但RestartPolicy说我们应该重新启动它。
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]:I0106 05:41:45.454658 819 kuberuntime_-manager.go:739]检查“kube-controller-manager-ip-172-20-49-150.us-west-1.compute.内部_-kube-system”(ef6f03ef0b14d853dd38e4c2a5f426dc)吊舱中容器“kube-controller-manager”的退避
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]:I0106 05:41:45.454781 819 kuberuntime_-manager.go:749]后退500秒重新启动失败的容器=kube控制器管理器pod=kube-controller-manager-ip-172-20-49-150.us-west-1.compute.内部_-kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]:E0106 05:41:45.454813 819 pod_-workers.go:182]同步pod EF6F03EF0B14D853D38E4C2A5F426DC时出错(“kube-controller-manager-ip-172-20-49-150.us-west-1.compute.internal__-kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)”),跳过:无法启动“kube控制器经理”的“集装箱”“使用CrashLoopBackOff:“后退5ms重新启动失败的容器=kube控制器管理器pod=kube-controller-manager-ip-172-20-49-150.us-west-1.compute.internal_kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)”
Jan 06 05:41:47 ip-172-20-49-150 kubelet[819]:I0106 05:41:47.432074 819容器。go:471]未能更新容器的统计信息“/kubepods/burstable/pod2a5faee9437283d8ac7f396d86d07a03/0F62EA06693A7D4AAF6702D8CA370F2D5D2F3C4FDEAB09AEDE15EA5311E47C”:无法确定目录的设备信息:/var/lib/docker/overlay/ce30183e915076727e708ed10b2ada4d55d1fe6d5c989c1cffc3e29cc00dad94:stat在/var/lib/docker/overlay/ce30183e915076727e708ed10b2ada4d55d1fe6d5c989c1cffc3e29cc00dad94上失败,错误:没有此类文件或目录,继续推送stats
看起来您已使用标志--unsecure bind address=127.0.0.1
启动了api服务器。主机上的8080端口不可用,因此无法启动
在127.0.0.1:8080上不安全地提供服务无法在127.0.0.1:8080上侦听:侦听tcp 127.0.0.1:8080:bind:address已在使用
我用新的K8s开发主节点替换了旧的K8s开发主节点,但仍然遇到了同样的问题,现在当将K8s主节点从c4.large垂直扩展到c4.xlarge时,它工作得很好!Kube API serveradmi的日志n@ip-172-20-49-150:/etc/kubernetes/manifests$sudo tail-f/var/log/kube-apiserver.log[restful]2020/01/06 07:35:10 log.go:33:[restful/swagger]映射到文件夹/swagger ui/I0106 07:35:11.593821 5跟踪。go:76]跟踪[1145457816]:“decodelist*[]api。