Docker Heapster无法从Kubernetes群集上的Kubelet获取容器统计信息
我已经根据指南在Ubuntu(trusty)上建立了一个Kubernetes集群,部署了一个DNS,并使用一个XDB后端和一个Grafana UI运行Heapster 除了Grafana之外,一切似乎都运行得很顺利,Grafana没有显示任何图形,只是在其图表中显示消息Docker Heapster无法从Kubernetes群集上的Kubelet获取容器统计信息,docker,kubernetes,heapster,Docker,Kubernetes,Heapster,我已经根据指南在Ubuntu(trusty)上建立了一个Kubernetes集群,部署了一个DNS,并使用一个XDB后端和一个Grafana UI运行Heapster 除了Grafana之外,一切似乎都运行得很顺利,Grafana没有显示任何图形,只是在其图表中显示消息无数据点: 在检查Docker容器日志后,我发现Heapster无法访问kubelet API(?),因此没有任何指标保存到XDB中: user@host:~$ docker logs e490a3ac10a8 I0701 07:
无数据点
:
在检查Docker容器日志后,我发现Heapster无法访问kubelet API(?),因此没有任何指标保存到XDB中:
user@host:~$ docker logs e490a3ac10a8
I0701 07:07:30.829745 1 heapster.go:65] /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb:8086
I0701 07:07:30.830082 1 heapster.go:66] Heapster version 1.2.0-beta.0
I0701 07:07:30.830809 1 configs.go:60] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0701 07:07:30.831284 1 configs.go:61] Using kubelet port 10255
E0701 07:09:38.196674 1 influxdb.go:209] issues while creating an InfluxDB sink: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp 10.0.0.223:8086: getsockopt: connection timed out, will retry on use
I0701 07:09:38.196919 1 influxdb.go:223] created influxdb sink with options: host:monitoring-influxdb:8086 user:root db:k8s
I0701 07:09:38.197048 1 heapster.go:92] Starting with InfluxDB Sink
I0701 07:09:38.197154 1 heapster.go:92] Starting with Metric Sink
I0701 07:09:38.228046 1 heapster.go:171] Starting heapster on port 8082
I0701 07:10:05.000370 1 manager.go:79] Scraping metrics start: 2016-07-01 07:09:00 +0000 UTC, end: 2016-07-01 07:10:00 +0000 UTC
E0701 07:10:05.008785 1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
I0701 07:10:05.009119 1 manager.go:152] ScrapeMetrics: time: 8.013178ms size: 0
I0701 07:11:05.001185 1 manager.go:79] Scraping metrics start: 2016-07-01 07:10:00 +0000 UTC, end: 2016-07-01 07:11:00 +0000 UTC
E0701 07:11:05.007130 1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
I0701 07:11:05.007686 1 manager.go:152] ScrapeMetrics: time: 5.945236ms size: 0
W0701 07:11:25.010298 1 manager.go:119] Failed to push data to sink: InfluxDB Sink
I0701 07:12:05.000420 1 manager.go:79] Scraping metrics start: 2016-07-01 07:11:00 +0000 UTC, end: 2016-07-01 07:12:00 +0000 UTC
E0701 07:12:05.002413 1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
I0701 07:12:05.002467 1 manager.go:152] ScrapeMetrics: time: 1.93825ms size: 0
E0701 07:12:12.309151 1 influxdb.go:150] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp 10.0.0.223:8086: getsockopt: connection timed out
I0701 07:12:12.351348 1 influxdb.go:201] Created database "k8s" on influxDB server at "monitoring-influxdb:8086"
I0701 07:13:05.001052 1 manager.go:79] Scraping metrics start: 2016-07-01 07:12:00 +0000 UTC, end: 2016-07-01 07:13:00 +0000 UTC
E0701 07:13:05.015947 1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
...
我在GitHub上发现了一些描述类似问题的问题,这些问题让我明白Heapster不是通过节点的环回访问kubelet,而是通过容器的环回访问kubelet本身。但是,我无法复制他们的解决方案:
github.com/kubernetes/heapster/issues/1183
您应该为Heapster pod使用主机网络,或者以节点具有常规名称而不是127.0.0.1的方式配置集群。当前的问题是节点名被解析为Heapster localhost。如果出现更多问题,请重新打开
-@皮奥斯
- 如何为我的Heapster pod启用“主机网络”
- 如何将群集/节点配置为使用常规名称而不是127.0.0.1
- 是否有办法通过在
中添加/修改kubelet的选项标志来解决此问题docker run
我尝试设置--hostname override=:10255:getsockopt:连接被拒绝
default
命名空间中运行Kubernetes API,在kube系统中运行Heapster
user@host:~$ kubectl get --all-namespaces pods
NAMESPACE NAME READY STATUS RESTARTS AGE
default k8s-etcd-127.0.0.1 1/1 Running 0 18h
default k8s-master-127.0.0.1 4/4 Running 1 18h
default k8s-proxy-127.0.0.1 1/1 Running 0 18h
kube-system heapster-lizks 1/1 Running 0 18h
kube-system influxdb-grafana-e0pk2 2/2 Running 0 18h
kube-system kube-dns-v10-4vjhm 4/4 Running 0 18h
操作系统:Ubuntu14.04.4LTS(可靠的)|
库伯内特斯:v1.2.5|
Docker:v1.11.2Heapster已经从Kubernetes获得了节点列表,现在正试图从每个节点上的kublete进程中提取统计数据(该进程有一个内置的收集节点统计数据的cAdvisor)。在本例中,只有一个节点,kubernetes知道它是127.0.0.1。这就是问题所在。Heapster容器正试图到达127.0.0.1的节点,该节点本身就是Heapster容器,当然,在Heapster容器中找不到要查询的kublete进程
要解决这个问题,需要做两件事
我们需要通过127.0.0.1环回网络地址以外的其他内容引用kublete worker节点(我们运行kubernetes的主机)
kublete进程需要接受来自新网络接口/地址的流量
假设您正在使用本地安装指南,并使用
hack/local-up-cluster.sh
更改引用kublete的主机名非常简单。您可以采用更详细的方法,但将其设置为eth0ip对我来说效果很好(ifconfig eth0)。缺点是,您需要一个eth0接口,这取决于DHCP,因此您的里程数可能会因方便程度而异
export HOSTNAME_OVERRIDE=10.0.2.15
让kublete进程接受来自任何网络接口的流量也同样简单
export KUBELET_HOST=0.0.0.0
为heapster配置提供以下参数以解决此问题
--source=kubernetes:所有名称空间中都会出现“kubernetes”服务,但指向相同的内容。我认为这不是你的问题。“本地”指南有点像玩具,我们不会对其运行自动测试,因此很可能这在本地不起作用。@EricTune非常感谢您的回答,至少我现在可以忽略假定的命名空间问题。但是现在正在使用一个无权限的服务帐户,在日志中输出“请求失败…”403“禁止”。我应该配置哪些权限?