谷歌云中的Kubernetes在容器创建状态下出现问题
我的GKE集群有问题，所有的POD都处于容器创建状态。运行kubectl get事件时，我看到以下错误：谷歌云中的Kubernetes在容器创建状态下出现问题,kubernetes,google-kubernetes-engine,Kubernetes,Google Kubernetes Engine,我的GKE集群有问题，所有的POD都处于容器创建状态。运行kubectl get事件时，我看到以下错误： Failed create pod sandbox: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded
Failed create pod sandbox: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Name: gke-aditum-k8scluster--pool-nodes-dev-500ebc8b-bgb6
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/fluentd-ds-ready=true
beta.kubernetes.io/instance-type=n1-standard-1
beta.kubernetes.io/os=linux
cloud.google.com/gke-nodepool=pool-nodes-dev
failure-domain.beta.kubernetes.io/region=southamerica-east1
failure-domain.beta.kubernetes.io/zone=southamerica-east1-a
kubernetes.io/hostname=gke-aditum-k8scluster--pool-nodes-dev-500ebc8b-bgb6
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp: Thu, 27 Sep 2018 20:27:47 -0300
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
KernelDeadlock False Fri, 28 Sep 2018 09:58:58 -0300 Thu, 27 Sep 2018 20:27:16 -0300 KernelHasNoDeadlock kernel has no deadlock
FrequentUnregisterNetDevice False Fri, 28 Sep 2018 09:58:58 -0300 Thu, 27 Sep 2018 20:32:18 -0300 UnregisterNetDevice node is functioning properly
NetworkUnavailable False Thu, 27 Sep 2018 20:27:48 -0300 Thu, 27 Sep 2018 20:27:48 -0300 RouteCreated NodeController create implicit route
OutOfDisk False Fri, 28 Sep 2018 09:59:03 -0300 Thu, 27 Sep 2018 20:27:47 -0300 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Fri, 28 Sep 2018 09:59:03 -0300 Thu, 27 Sep 2018 20:27:47 -0300 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Fri, 28 Sep 2018 09:59:03 -0300 Thu, 27 Sep 2018 20:27:47 -0300 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Fri, 28 Sep 2018 09:59:03 -0300 Thu, 27 Sep 2018 20:27:47 -0300 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Fri, 28 Sep 2018 09:59:03 -0300 Thu, 27 Sep 2018 20:28:07 -0300 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 10.0.0.2
ExternalIP:
Hostname: gke-aditum-k8scluster--pool-nodes-dev-500ebc8b-bgb6
Capacity:
cpu: 1
ephemeral-storage: 98868448Ki
hugepages-2Mi: 0
memory: 3787608Ki
pods: 110
Allocatable:
cpu: 940m
ephemeral-storage: 47093746742
hugepages-2Mi: 0
memory: 2702168Ki
pods: 110
System Info:
Machine ID: 1e8e0ecad8f5cc7fb5851bc64513d40c
System UUID: 1E8E0ECA-D8F5-CC7F-B585-1BC64513D40C
Boot ID: 971e5088-6bc1-4151-94bf-b66c6c7ee9a3
Kernel Version: 4.14.56+
OS Image: Container-Optimized OS from Google
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://17.3.2
Kubelet Version: v1.10.7-gke.2
Kube-Proxy Version: v1.10.7-gke.2
PodCIDR: 10.0.32.0/24
ProviderID: gce://aditumpay/southamerica-east1-a/gke-aditum-k8scluster--pool-nodes-dev-500ebc8b-bgb6
Non-terminated Pods: (11 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system event-exporter-v0.2.1-5f5b89fcc8-xsvmg 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system fluentd-gcp-scaler-7c5db745fc-vttc9 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system fluentd-gcp-v3.1.0-sz8r8 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system heapster-v1.5.3-75486b456f-sj7k8 138m (14%) 138m (14%) 301856Ki (11%) 301856Ki (11%)
kube-system kube-dns-788979dc8f-99xvh 260m (27%) 0 (0%) 110Mi (4%) 170Mi (6%)
kube-system kube-dns-788979dc8f-9sz2b 260m (27%) 0 (0%) 110Mi (4%) 170Mi (6%)
kube-system kube-dns-autoscaler-79b4b844b9-6s8x2 20m (2%) 0 (0%) 10Mi (0%) 0 (0%)
kube-system kube-proxy-gke-aditum-k8scluster--pool-nodes-dev-500ebc8b-bgb6 100m (10%) 0 (0%) 0 (0%) 0 (0%)
kube-system kubernetes-dashboard-598d75cb96-6nhcd 50m (5%) 100m (10%) 100Mi (3%) 300Mi (11%)
kube-system l7-default-backend-5d5b9874d5-8wk6h 10m (1%) 10m (1%) 20Mi (0%) 20Mi (0%)
kube-system metrics-server-v0.2.1-7486f5bd67-fvddz 53m (5%) 148m (15%) 154Mi (5%) 404Mi (15%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 891m (94%) 396m (42%)
memory 817952Ki (30%) 1391392Ki (51%)
Events: <none>
Name: aditum-payment-7d966c494c-wpk2t
Namespace: default
Node: gke-aditum-k8scluster--pool-nodes-dev-500ebc8b-m7bz/10.0.0.3
Start Time: Thu, 27 Sep 2018 20:30:47 -0300
Labels: io.kompose.service=aditum-payment
pod-template-hash=3852270507
Annotations: kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container aditum-payment
Status: Pending
IP:
Controlled By: ReplicaSet/aditum-payment-7d966c494c
Containers:
aditum-payment:
Container ID:
Image: gcr.io/aditumpay/aditumpaymentwebapi:latest
Image ID:
Port: 5000/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 100m
Environment:
CONNECTIONSTRING: <set to the key 'CONNECTIONSTRING' of config map 'aditum-payment-config'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-qsc9k (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-qsc9k:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-qsc9k
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 3m (x1737 over 13h) kubelet, gke-aditum-k8scluster--pool-nodes-dev-500ebc8b-m7bz Failed create pod sandbox: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
名称：aditum-payment-7d966c494c-wpk2t
名称空间：默认值
节点：gke-aditum-k8scluster--pool-NOTES-dev-500ebc8b-m7bz/10.0.0.3
开始时间：2018年9月27日星期四20:30:47-0300
标签：io.kompose.service=aditum付款
pod模板哈希=3852270507
注释：kubernetes.io/limit ranger=LimitRanger插件集：cpu请求容器aditum付款
状态：待定
知识产权：
控制人：ReplicaSet/aditum-payment-7d966c494c
容器：
预付款：
容器ID:
图片：gcr.io/aditumpay/aditumpaymentwebapi:latest
图像ID:
端口：5000/TCP
主机端口：0/TCP
国家：等待
原因：集装箱制造
就绪：错误
重新启动计数：0
请求：
中央处理器：100米
环境：
CONNECTIONSTRING:可选：false
挂载：
/来自default-token-qsc9k（ro）的var/run/secrets/kubernetes.io/serviceCount
条件：
类型状态
初始化为True
准备错误
播客预定为真
卷数：
default-token-qsc9k：
类型：Secret（由Secret填充的卷）
SecretName:default-token-qsc9k
可选：false
QoS等级：Burstable
节点选择器：
容差：node.kubernetes.io/未就绪：不执行300秒
node.kubernetes.io/不可访问：不执行300秒
活动：
从消息中键入原因年龄
---- ------ ---- ---- -------
警告FailedCreatePodSandBox 3m（x1737超过13小时）kubelet，gke-aditum-k8scluster--pool-NOTES-dev-500ebc8b-m7bz创建pod sandbox失败：rpc错误：代码=未知描述=来自守护程序的错误响应：Gethttps://k8s.gcr.io/v2/: net/http:在等待连接时取消请求（在等待标头时超过Client.Timeout）
谢谢 除了节点的描述之外，它还取决于从何处启动节点 如或中所述，这在中国是行不通的 这种情况下的解决方法是找到另一个源，将其拉入并标记：
minikube ssh \
"docker pull registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
docker tag registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0 gcr.io/google_containers/pause-amd64:3.0"
很抱歉，回复时间过长。这是一个非常愚蠢的问题。在我获得谷歌云支持后，我注意到我的NAT机器工作不正常。私密通道经过我的房间。谢谢大家的帮助。你能发布
kubectl描述节点吗？还有kubectl description pod您是否已按此处所述为您的子网启用了私有google访问：？@Jukka是的，我启用了。集群专有网络是本地的吗？我刚刚测试了将VPC本地私有集群创建成一个子网，并启用了私有google访问，所有kube系统吊舱都正常启动。是的，我知道。我做到了。到昨天为止，集群一直运转良好。这个问题是突然出现的。我正在巴西发布它们。所以我认为这个问题不适用。@ArturFernandes，你能对那个图像做一个简单的docker拉动吗？在我的例子中，我使用的是私人GKE。所以我使用GCR。在子网络描述中，我可以激活一个名为PrivateGoogleAccess的属性。我试图提取的图像都来自GCR。哦，对不起，我可以从我的私人注册表中进行简单的docker提取。@ArturFernandes好的，所以这个答案不适用于您的情况。我将把它留在这里，以防它与另一个读者的问题相匹配。很好地发现，+1！
Name: aditum-payment-7d966c494c-wpk2t
Namespace: default
Node: gke-aditum-k8scluster--pool-nodes-dev-500ebc8b-m7bz/10.0.0.3
Start Time: Thu, 27 Sep 2018 20:30:47 -0300
Labels: io.kompose.service=aditum-payment
pod-template-hash=3852270507
Annotations: kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container aditum-payment
Status: Pending
IP:
Controlled By: ReplicaSet/aditum-payment-7d966c494c
Containers:
aditum-payment:
Container ID:
Image: gcr.io/aditumpay/aditumpaymentwebapi:latest
Image ID:
Port: 5000/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 100m
Environment:
CONNECTIONSTRING: <set to the key 'CONNECTIONSTRING' of config map 'aditum-payment-config'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-qsc9k (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-qsc9k:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-qsc9k
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 3m (x1737 over 13h) kubelet, gke-aditum-k8scluster--pool-nodes-dev-500ebc8b-m7bz Failed create pod sandbox: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
minikube ssh \
"docker pull registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
docker tag registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0 gcr.io/google_containers/pause-amd64:3.0"