Kubernetes 卷“的MountVolume.SetUp失败;“策略适配器机密”:不能';t传播对象缓存:等待条件时超时

Kubernetes 卷“的MountVolume.SetUp失败;“策略适配器机密”:不能';t传播对象缓存:等待条件时超时,kubernetes,kubectl,istio,Kubernetes,Kubectl,Istio,使用kubeadm创建了一个两节点集群 已安装的istio 1.1.11 kubectl版本 Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5

使用kubeadm创建了一个两节点集群

已安装的istio 1.1.11

kubectl版本

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
执行istio文档中给出的命令

$
用于安装/kubernetes/helm/istio init/files/crd*yaml中的i;kubectl是否适用-f$i;完成

$
kubectl apply-f install/kubernetes/istio demo.yaml

创建了服务

$
kubectl获取吊舱-istio系统

遥测和策略pod状态变为CrashLoopBackOff状态

istio-policy-648b5f5bb5-dv5np                 1/2        **CrashLoopBackOff**      5          2m52s

istio-telemetry-57946b8569-9m7gd           1/2     **CrashLoopBackOff**   5          2m52s
描述pod时,出现以下错误

 Warning  FailedMount  2m16s (x2 over 2m18s)  kubelet, ip-xxx-xxx-xxx-xxx  MountVolume.SetUp failed for volume "policy-adapter-secret" : couldn't propagate object cache: timed out waiting for the condition
已尝试重新启动VM,已重新启动docker服务。这没有帮助

由于上述错误,pod反复尝试重新启动,然后崩溃


需要您帮助解决此问题

如果您的节点没有足够的内存运行Istio,这些混音器服务可能会崩溃。越来越多的人使用诸如安装Istio(和其他服务网格)之类的工具,因为它会突出显示内存之类的争用点。在部署
istio-demo
istio-demo-auth
配置文件时,您需要确保每个节点都有一个配置文件(特别是,如果istio控制平面仅部署到一个节点).

在网络中,您可以发现许多与
无法传播对象缓存相关的问题:等待条件时超时。
Github上已经有公开发行的版本-

作为解决此问题的众多步骤之一,请尝试:

  • 重新启动群集
  • 重新启动docker deamon
  • 更新你的操作系统
在与ISTIO有关的案例中,我在Kubeadm、Minikube和GKE上试用过。 在所有情况下,
istio policy XXX XXX XXX
istio遥测XXX XXX
由于活动性问题失败而重新启动

telemetry example
---
  Warning  Unhealthy  8m49s (x9 over 9m29s)  kubelet, gke-istio-default-pool-c41459f8-zbhn  Liveness probe failed: Get http://10.56.0.6:15014/version: dial tcp 10.56.0.6:15014: connect: connection refused
  Normal   Killing    8m49s (x3 over 9m19s)  kubelet, gke-istio-default-pool-c41459f8-zbhn  Killing container with id docker://mixer:Container failed liveness probe.. Container will be killed and recreated.

policy example
---
  Warning  Unhealthy  7m28s (x9 over 8m8s)   kubelet, gke-istio-default-pool-c41459f8-3c6d  Liveness probe failed: Get http://10.56.2.6:15014/version: dial tcp 10.56.2.6:15014: connect: connection refused
  Normal   Killing    7m28s (x3 over 7m58s)  kubelet, gke-istio-default-pool-c41459f8-3c6d  Killing container with id docker://mixer:Container failed liveness probe.. Container will be killed and recreated.
即使在文档示例中,您也可以看到遥测和策略被重新启动了2次

在验证了两个yaml(istio-demo.yaml和istio-demo auth.yaml)后,我发现遥测和策略
部署
已将liveness probe设置为5

livenessProbe:
          httpGet:
            path: /version
            port: 15014
          initialDelaySeconds: 5
          periodSeconds: 5
如果您将从
istio telemetry
pod在混合器容器上使用kubectl日志,您可能会看到一些错误,如

2019-07-18T15:16:01.887334Z     info    pickfirstBalancer: HandleSubConnStateChange: 0xc420751a80, CONNECTING
...
2019-07-18T15:16:21.887741Z     info    pickfirstBalancer: HandleSubConnStateChange: 0xc420751a80, TRANSIENT_FAILURE
2019-07-18T15:16:21.887862Z     error   mcp     Failed to create a new MCP sink stream: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 10.0.25.27:9901: i/o timeout"
...
2019-07-18T15:16:44.282027Z     info    pickfirstBalancer: HandleSubConnStateChange: 0xc420751a80, CONNECTING
2019-07-18T15:16:44.287281Z     info    pickfirstBalancer: HandleSubConnStateChange: 0xc420751a80, READY
2019-07-18T15:16:44.888794Z     info    mcp     (re)trying to establish new MCP sink stream
2019-07-18T15:16:44.888922Z     info    mcp     New MCP sink stream created
因此,简而言之,两种(遥测和策略)部署中的混合器容器都需要大约44秒来建立所有连接

如果在两种部署中都将
initialDelaySeconds:
更改为60秒,则不应重新启动POD,原因是liveness probe


希望这有帮助

类似的地方:检查版本。这些版本是1.15版本,将CPU增加到8个vCore和32 GB RAM内存。重新部署istio演示。但问题依然存在。你能帮我做些调试吗?这解决了我的问题,我试着用不到4Gi的内存运行cassandra helm。谢谢!“针对卷“pvc-edd053ea-6eed-4763-85d5”的MountVolume.MountDevice失败:rpc错误:code=Internal desc=安装失败:退出状态255 cmd:'mount-t ext4/dev/disk/by id/scsi-0DO\u volume\u pvc-edd053ea-6eed-4763-85d5/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-edd053ea-6eed-4763-85d5-046d7bb6238c/globalmount”输出:”挂载:在/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-edd053ea-6eed-4763-85d5上挂载/dev/disk/by-id/scsi-0DO\u Volume\u pvc-edd053ea-6eed-4763-85d5/globalmount失败:资源繁忙\n“\uuuuuuu删除状态集po时出现此错误,重新启动节点解决了此问题。我使用了digitalocean csi