Kubernetes devk8s主机显示额外负载,导致无法获取POD的输出

Kubernetes devk8s主机显示额外负载,导致无法获取POD的输出,kubernetes,kubectl,kubelet,Kubernetes,Kubectl,Kubelet,我的开发人员k8s主机显示额外负载,导致无法获取POD的输出: admin@ip-172-20-49-150:~$kubectl获得订单-n cog阶段 ^C admin@ip-172-20-49-150:~$ admin@ip-172-20-49-150:~$top top - 04:36:52 up 2 min, 2 users, load average: 14.39, 4.43, 1.55 Tasks: 140 total, 2 running, 138 sleeping,

我的开发人员k8s主机显示额外负载,导致无法获取POD的输出:

admin@ip-172-20-49-150:~$kubectl获得订单-n cog阶段

^C
admin@ip-172-20-49-150:~$
admin@ip-172-20-49-150:~$top

top - 04:36:52 up 2 min,  2 users,  load average: 14.39, 4.43, 1.55
Tasks: 140 total,   2 running, 138 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.2 sy,  0.0 ni,  0.0 id, 99.6 wa,  0.0 hi,  0.0 si,  0.2 st
KiB Mem:   3857324 total,  3778024 used,    79300 free,      192 buffers
KiB Swap:        0 total,        0 used,        0 free.    15680 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   32 root      20   0       0      0      0 S   2.4  0.0   0:03.75 kswapd0
 1263 root      20   0   97388  19036      0 S   1.3  0.5   0:01.06 kube-controller
 1224 root      20   0   28764  11380      0 S   0.7  0.3   0:01.86 etcd
 1358 root      20   0   46192  10608      0 S   0.7  0.3   0:00.69 kube-scheduler
 1243 root      20   0  372552 343024      0 S   0.6  8.9   0:10.51 etcd
  695 root      20   0  889180  52352      0 S   0.4  1.4   0:05.34 dockerd
  752 root      20   0  205800  13756      0 S   0.4  0.4   0:00.56 protokube
  816 root      20   0  449964  30804      0 S   0.4  0.8   0:02.26 kubelet
 1247 root      20   0 3207664 2.856g      0 S   0.4 77.6   0:55.90 kube-apiserver
 1279 root      20   0   40848   8900      0 S   0.4  0.2   0:00.46 kube-proxy
    1 root      20   0   28788   1940      0 R   0.2  0.1   0:02.06 systemd
  157 root       0 -20       0      0      0 S   0.2  0.0   0:00.06 kworker/1:1H
 1562 admin     20   0   78320   1092      0 S   0.2  0.0   0:00.04 sshd
 1585 admin     20   0   23660    540      0 R   0.2  0.0   0:00.11 top
 1758 admin     20   0   33512    320     32 D   0.2  0.0   0:00.04 kubectl
 1779 root      20   0   39368    436      0 D   0.2  0.0   0:00.01 docker-containe
请让我知道如何解决此问题

在主机上更新kubelet日志: admin@ip-172-20-49-150:~$journalctl-u kubelet-f

Jan 06 05:41:44 ip-172-20-49-150 kubelet[819]: E0106 05:41:44.454586     819 pod_workers.go:182] Error syncing pod 685c903f9066f69a2e17c802cb043ed6 ("etcd-server-events-ip-172-20-49-150.us-west-1.compute.internal_kube-system(685c903f9066f69a2e17c802cb043ed6)"), skipping: failed to "StartContainer" for "etcd-container" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=etcd-container pod=etcd-server-events-ip-172-20-XX-XXX.us-west-1.compute.internal_kube-system(685c903f906b043ed6)"
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]: I0106 05:41:45.454266     819 kuberuntime_manager.go:500] Container {Name:kube-controller-manager Image:gcr.io/google_containers/kube-controller-manager:v1.8.4 Command:[/bin/sh -c /usr/local/bin/kube-controller-manager --allocate-node-cidrs=true --attach-detach-reconcile-sync-period=1m0s --cloud-provider=aws --cluster-cidr=100.96.0.0/11 --cluster-name=uw1b.k8s.ops.goldenratstud.io --cluster-signing-cert-file=/srv/kubernetes/ca.crt --cluster-signing-key-file=/srv/kubernetes/ca.key --configure-cloud-routes=true --kubeconfig=/var/lib/kube-controller-manager/kubeconfig --leader-elect=true --root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key --use-service-account-credentials=true --v=2 2>&1 | /bin/tee -a /var/log/kube-controller-manager.log] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:<nil>} s:100m Format:DecimalSI}]} VolumeMounts:[{Name:etcssl ReadOnly:true MountPath:/etc/ssl SubPath: MountPropagation:<nil>} {Name:etcpkitls ReadOnly:true MountPath:/etc/pki/tls SubPath: MountPropagation:<nil>} {Name:etcpkica-trust ReadOnly:true MountPath:/etc/pki/ca-trust SubPath: MountPropagation:<nil>} {Name:usrsharessl ReadOnly:true MountPath:/usr/share/ssl SubPath: MountPropagation:<nil>} {Name:usrssl ReadOnly:true MountPath:/usr/ssl SubPath: MountPropagation:<nil>} {Name:usrlibssl ReadOnly:true MountPath:/usr/lib/ssl SubPath: MountPropagation:<nil>} {Name:usrlocalopenssl ReadOnly:true MountPath:/usr/local/openssl SubPath: MountPropagation:<nil>} {Name:varssl ReadOnly:true MountPath:/var/ssl SubPath: MountPropagation:<nil>} {Name:etcopenssl ReadOnly:true MountPath:/etc/openssl SubPath: MountPropagation:<nil>} {Name:srvkube ReadOnly:true MountPath:/srv/kubernetes SubPath: MountPropagation:<nil>} {Name:logfile ReadOnly:false MountPath:/var/log/kube-controller-manager.log SubPath: MountPropagation:<nil>} {Name:varlibkcm ReadOnly:true MountPath:/var/lib/kube-controller-manager SubPath: MountPropagation:<nil>}] Live
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]: nessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:10252,Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]: I0106 05:41:45.454658     819 kuberuntime_manager.go:739] checking backoff for container "kube-controller-manager" in pod "kube-controller-manager-ip-172-20-49-150.us-west-1.compute.internal_kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)"
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]: I0106 05:41:45.454781     819 kuberuntime_manager.go:749] Back-off 5m0s restarting failed container=kube-controller-manager pod=kube-controller-manager-ip-172-20-49-150.us-west-1.compute.internal_kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]: E0106 05:41:45.454813     819 pod_workers.go:182] Error syncing pod ef6f03ef0b14d853dd38e4c2a5f426dc ("kube-controller-manager-ip-172-20-49-150.us-west-1.compute.internal_kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)"), skipping: failed to "StartContainer" for "kube-controller-manager" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kube-controller-manager pod=kube-controller-manager-ip-172-20-49-150.us-west-1.compute.internal_kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)"
Jan 06 05:41:47 ip-172-20-49-150 kubelet[819]: I0106 05:41:47.432074     819 container.go:471] Failed to update stats for container "/kubepods/burstable/pod2a5faee9437283d8ac7f396d86d07a03/0f62ea06693a7d4aaf6702d8ca370f2d5d2f1f3c4fdeab09aede15ea5311e47c": unable to determine device info for dir: /var/lib/docker/overlay/ce30183e915076727e708ed10b2ada4d55d1fe6d5c989c1cffc3e29cc00dad94: stat failed on /var/lib/docker/overlay/ce30183e915076727e708ed10b2ada4d55d1fe6d5c989c1cffc3e29cc00dad94 with error: no such file or directory, continuing to push stats
Jan 06 05:41:44 ip-172-20-49-150 kubelet[819]:E0106 05:41:44.454586 819 pod_-U-workers.go:182]错误同步pod 685c903f9066f69a2e17c802cb043ed6(“etcd-server-events-ip-172-20-49-150.us-west-1.compute.internal__-kube-system(685C903F9066F69A27C802CB043ED6)”),跳过:无法为“etcd容器”启动崩溃:“后退5ms重新启动失败的容器=etcd容器pod=etcd-server-events-ip-172-20-XX-XXX.us-west-1.compute.internal_kube-system(685c903f906b043ed6)”
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]:I0106 05:41:45.454266 819 kuberuntime_manager.go:500]容器{名称:kube控制器管理器图像:gcr.io/google_容器/kube控制器管理器:v1.8.4命令:[/bin/sh-c/usr/local/bin/kube controller manager--allocate node cidrs=true--attach detach-reconcil-sync period=1mos--cloud provider=aws--cluster cidr=100.96.0.0/11--cluster name=uw1b.k8s.ops.goldenratstud.io--cluster signing cert文件=/srv/kubernetes/ca.crt--cluster签名密钥文件=/srv/kubernetes/ca.key--configure cloud routes=--kubeconfig=/var/lib/kube controller manager/kubeconfig--leader elect=true--root ca file=/srv/kubernetes/ca.crt--service account private key file=/srv/kubernetes/server.key--use service account credentials=true--v=2>&1 |/bin/tee-a/var/log/kube controller manager.log]参数:[]工作目录:端口:[]环境:[]资源:[限制:映射[]请求:map[cpu:{i:{value:100 scale:-3}d:{Dec:}s:100m格式:小数点SI}]VolumeMounts:[{Name:etcssl ReadOnly:true MountPath:/etc/ssl子路径:MountPropagation:}{Name:etcpkitls ReadOnly:true MountPath:/etc/pki/tls子路径:mountpagation:}{Name:etcpkica trust ReadOnly:true MountPath:/etc/pki/ca trust子路径:mountpagation:}{Name:usrsharessl ReadOnly:true MountPath:/usr/share/ssl子路径:mountprogation:}{Name:usrssl ReadOnly:true MountPath:/usr/ssl子路径:mountprogation:}{Name:usrlibssl ReadOnly:true mountprogation:}{Name:usrlocalopenssl ReadOnly:true mountprogation:/usr/local/openssl子路径:mountprogation:}{Name:varssl ReadOnly:true MountPath:/var/ssl子路径:MountPropagation:}{Name:etcopenssl ReadOnly:true MountPath:/etc/openssl子路径:MountPropagation:}{Name:srvkube ReadOnly:true MountPath:/srv/kubernetes子路径:mountpagation:}{Name:logfile ReadOnly:false MountPath:/var/log/kube-controller-manager.log子路径:mountpagation:}{Name:varlibkcm ReadOnly:true MountPath:/var/lib/kube控制器管理器子路径:mountprogation:}]Live
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]:nessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,端口:10252,主机:127.0.0.1,方案:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodsSeconds:10,SuccessThreshold:1,FailureReshold:3,}ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}已死亡,但RestartPolicy说我们应该重新启动它。
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]:I0106 05:41:45.454658 819 kuberuntime_-manager.go:739]检查“kube-controller-manager-ip-172-20-49-150.us-west-1.compute.内部_-kube-system”(ef6f03ef0b14d853dd38e4c2a5f426dc)吊舱中容器“kube-controller-manager”的退避
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]:I0106 05:41:45.454781 819 kuberuntime_-manager.go:749]后退500秒重新启动失败的容器=kube控制器管理器pod=kube-controller-manager-ip-172-20-49-150.us-west-1.compute.内部_-kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)
Jan 06 05:41:45 ip-172-20-49-150 kubelet[819]:E0106 05:41:45.454813 819 pod_-workers.go:182]同步pod EF6F03EF0B14D853D38E4C2A5F426DC时出错(“kube-controller-manager-ip-172-20-49-150.us-west-1.compute.internal__-kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)”),跳过:无法启动“kube控制器经理”的“集装箱”“使用CrashLoopBackOff:“后退5ms重新启动失败的容器=kube控制器管理器pod=kube-controller-manager-ip-172-20-49-150.us-west-1.compute.internal_kube-system(ef6f03ef0b14d853dd38e4c2a5f426dc)”
Jan 06 05:41:47 ip-172-20-49-150 kubelet[819]:I0106 05:41:47.432074 819容器。go:471]未能更新容器的统计信息“/kubepods/burstable/pod2a5faee9437283d8ac7f396d86d07a03/0F62EA06693A7D4AAF6702D8CA370F2D5D2F3C4FDEAB09AEDE15EA5311E47C”:无法确定目录的设备信息:/var/lib/docker/overlay/ce30183e915076727e708ed10b2ada4d55d1fe6d5c989c1cffc3e29cc00dad94:stat在/var/lib/docker/overlay/ce30183e915076727e708ed10b2ada4d55d1fe6d5c989c1cffc3e29cc00dad94上失败,错误:没有此类文件或目录,继续推送stats

看起来您已使用标志
--unsecure bind address=127.0.0.1
启动了api服务器。主机上的8080端口不可用,因此无法启动


在127.0.0.1:8080上不安全地提供服务无法在127.0.0.1:8080上侦听:侦听tcp 127.0.0.1:8080:bind:address已在使用
我用新的K8s开发主节点替换了旧的K8s开发主节点,但仍然遇到了同样的问题,现在当将K8s主节点从c4.large垂直扩展到c4.xlarge时,它工作得很好!

Kube API serveradmi的日志n@ip-172-20-49-150:/etc/kubernetes/manifests$sudo tail-f/var/log/kube-apiserver.log[restful]2020/01/06 07:35:10 log.go:33:[restful/swagger]映射到文件夹/swagger ui/I0106 07:35:11.593821 5跟踪。go:76]跟踪[1145457816]:“decodelist*[]api。