Kubernetes 容器优化操作系统k8s节点丢失google-ip-forwarding-daemon.service ip

版本：Kubernetes

v1.9.2

在GCE中运行（kube up.sh）

容器优化操作系统

10032.88.0

---

症状：我们的COS节点在其

谷歌ip转发守护进程中显示以下内容。服务
日志：

Mar 07 13:43:28 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.154.243.135'].
Mar 07 13:43:45 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.198.28.188'].
Mar 07 13:44:01 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.171.xxx'].
Mar 07 13:44:17 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'130.211.120.8'].
Mar 07 13:44:37 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.198.52.xxx'].
Mar 07 13:44:53 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.17.xx'].
Mar 07 13:45:10 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.155.181.xx'].
Mar 07 13:45:26 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.255.xxx'].
Mar 07 13:45:43 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.166.xxx'].
Mar 07 13:45:59 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] by adding None and removing [u'146.148.61.xx'].
Mar 07 13:46:19 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.197.104.xxx'].
Mar 07 13:46:41 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.198.172.xxx'].
Mar 07 13:46:58 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.198.162.xxx'].
Mar 07 13:47:14 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.155.131.xxx'].
Mar 07 13:47:34 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'130.211.238.xxx'] by adding None and removing [u'104.198.131.xxx'].
Mar 07 13:47:53 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'130.211.238.xxx'] by adding None and removing [u'104.154.72.xxx'].
Mar 07 13:48:10 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'130.211.238.xxx'] to [u'130.211.238.xxx'] by adding None and removing [u'104.154.20.xx'].
Mar 07 13:48:26 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'130.211.238.xxx'] to None by adding None and removing [u'130.211.238.xxx'].

当这种情况发生时，运行的吊舱通常会继续运行，但有时会变得未知并被重新安排。任何新计划的POD都会被困在ContainerCreating中，并出现以下错误：

创建pod沙盒失败。
&
守护进程的错误响应：未找到网络

除了重新启动实例之外，我们还没有找到解决此症状的方法。
Kubernetes 1.2和1.9.2之间的版本太多，为什么您要尝试创建一个非常旧的版本，然后对其进行升级？我遵循相同的Kubernetes文档，得到了一个运行1.9.3的集群，没有任何问题。您应该尝试同样的方法，并验证您描述的问题是否仍然存在

此外，该图像不是最新版本。你也应该试着用跑步代替跑步

在这个问题的评论中，建议禁用docker。这个有见地的建议可以缓解症状。阅读了live restore文档后，有人能确认启用docker的live restore（在kubernetes生态系统中）所获得的好处之一是在实例重启之间保留运行的POD吗？如果是这样，这种重启之间的保护行为是否是在kube中启用实时恢复的主要好处？我们正在运行一些禁用实时还原的节点作为测试，但是我正在尝试发现禁用此功能的全部影响。
您能发布创建群集的完整命令吗？您使用的是什么？此群集是使用kube-up.sh（）创建的kubernetes 1.2群集，并使用upgrade.sh（）升级到kubernetes 1.9.2在主要版本之间升级。如果您对我的kube env或特定的kube apiserver/kubelet参数有任何疑问，我也很乐意提供。您的网络驱动程序存在问题。尝试删除/etc/cni/net.d并重新启动kubelet或尝试设置另一个驱动程序
kubectl apply-f“https://cloud.weave.works/k8s/net?k8s-version=$（kubectl version | base64 | tr-d'\n'）“
您的节点上是否启用了docker的实时还原？如果是这样，请尝试禁用它，并在再次遇到相同问题时重新启动docker守护程序，以查看节点是否可以从中恢复。@Yu JuHong holy moly，成功了！您知道在k8s节点上禁用实时还原的影响吗？我们这样做会失去一些稳定性吗？当然，我明白你的意思。事实上，我注意到kube-up.sh-style 1.9.2集群和我的upgrade.sh-upgraded 1.2->1.9.2集群之间有很多不同之处。它已以这种方式升级，因为它是一个具有大量流量/工作负载的生产群集。是的，即使docker守护程序重新启动，live restore也允许容器继续运行。如果您希望启用live restore，可以等待并升级到1.10，或使用此修补程序修复配置：
Mar 12 13:21:04 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from None to [u'104.155.131.xxx'] by adding [u'104.155.131.xxx'] and removing None.
Mar 12 13:21:21 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.155.131.xxx'] to [u'104.154.72.xxx', u'104.155.131.xxx'] by adding [u'104.154.72.xxx'] and removing None.
Mar 12 13:21:37 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.72.xxx', u'104.155.131.xxx'] to [u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx'] by adding [u'104.198.131.xxx'] and removing None.
Mar 12 13:21:53 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx'] by adding [u'104.154.20.xx'] and removing None.
Mar 12 13:22:10 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'130.211.238.xxx'] and removing None.
Mar 12 13:22:26 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'104.154.243.xxx'] and removing None.
Mar 12 13:22:43 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'104.198.52.xxx'] and removing None.
Mar 12 13:22:59 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'104.198.28.xxx'] and removing None.
Mar 12 13:23:15 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'104.197.171.xxx'] and removing None.
Mar 12 13:23:28 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'130.211.120.x'] and removing None.
Mar 12 13:23:42 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'104.197.255.xxx'] and removing None.
Mar 12 13:23:58 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.17.xx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'104.197.17.xx'] and removing None.
Mar 12 13:24:15 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.17.xx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'104.155.181.xx'] and removing None.
Mar 12 13:24:31 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'104.197.166.xxx'] and removing None.
Mar 12 13:24:48 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] by adding [u'146.148.61.xx'] and removing None.
Mar 12 13:25:01 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] by adding [u'104.197.104.xxx'] and removing None.
Mar 12 13:25:17 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.172.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] by adding [u'104.198.172.xxx'] and removing None.
Mar 12 13:25:34 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.172.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] by adding [u'104.198.162.xxx'] and removing None.