Ssl kubelet服务可以'；由于错误net/http:TLS握手超时，无法使用https访问端口6443处的kube apiserver

我正在通过集群API在openstack的顶部配置一个工作负载集群，其中有一个控制平面节点和一个工作节点。但是，kubernetes控制平面未能在控制平面节点中正确启动

我可以看到kube apiserver不断退出并重新创建：

ubuntu@ubu1910-medflavor-nolb3-control-plane-nh4hf:~$ sudo crictl --runtime-endpoint /run/containerd/containerd.sock ps -a
CONTAINER           IMAGE               CREATED              STATE               NAME                      ATTEMPT             POD ID
a729fdd387b0a       90d27391b7808       About a minute ago   Running             kube-apiserver            74                  88de61a0459f6
38b54a71cb0aa       90d27391b7808       3 minutes ago        Exited              kube-apiserver            73                  88de61a0459f6
24573a1c5adc5       b0f1517c1f4bb       18 minutes ago       Running             kube-controller-manager   4                   cc113aaae13b5
a2072b64cca1a       b0f1517c1f4bb       29 minutes ago       Exited              kube-controller-manager   3                   cc113aaae13b5
f26a531972518       d109c0821a2b9       5 hours ago          Running             kube-scheduler            1                   df1d15fd61a8f
a91b4c0ce9e27       303ce5db0e90d       5 hours ago          Running             etcd                      1                   16e1f0f5bb543
1565a1a7dedec       303ce5db0e90d       5 hours ago          Exited              etcd                      0                   16e1f0f5bb543
35ae23eb64f11       d109c0821a2b9       5 hours ago          Exited              kube-scheduler            0                   df1d15fd61a8f
ubuntu@ubu1910-medflavor-nolb3-control-plane-nh4hf:~$

从kube apiserver容器的日志中，我可以看到“172.24.4.159:50812:EOF中的http:TLS握手错误”：

从系统日志中，我可以看到为IP 172.24.4.159签名的apiserver服务证书：

ubuntu@ubu1910-medflavor-nolb3-control-plane-nh4hf:~$ grep "apiserver serving cert is signed for DNS names" /var/log/syslog 
Apr 16 15:25:56 ubu1910-medflavor-nolb3-control-plane-nh4hf cloud-init[652]: [certs] apiserver serving cert is signed for DNS names [ubu1910-medflavor-nolb3-control-plane-nh4hf kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.6.0.9 172.24.4.159]

从syslog中，我还可以看到kubelet服务由于“net/http:TLS握手超时”而无法访问apiserver：

我还尝试使用curl访问apiserver，我发现：

ubuntu@ubu1910-medflavor-nolb3-control-plane-nh4hf:~$ curl http://172.24.4.159:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-ubu1910-medflavor-nolb3-control-plane-nh4hf
Client sent an HTTP request to an HTTPS server.

ubuntu@ubu1910-medflavor-nolb3-control-plane-nh4hf:~$ curl https://172.24.4.159:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-ubu1910-medflavor-nolb3-control-plane-nh4hf
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
ubuntu@ubu1910-medflavor-nolb3-control-plane-nh4hf:~$

---

kube apiserver的证书有问题吗？知道如何继续进行故障排除吗？

如果您想查看kube api SSL证书的详细信息，可以使用

curl-k-vhttps://172.24.4.159:6443

或

openssl s_客户端-连接172.24.4.159:6443

您没有提到如何设置证书。kubernetes中的SSL非常复杂，手动设置证书和所有通信可能非常痛苦。这就是为什么现在人们使用

kubeadm

TLDR：您必须确保所有证书都由

/etc/kubernetes/pki/ca.crt

签名

既然您提到了“单节点”，我想Kubelet是作为SystemD单元在同一台服务器上运行的？kube api容器是如何启动的？因为您在

/etc/kubernetes/manifests

中有pod定义，所以由Kubelet进程本身执行

实际上，在

kubelet

和

kube-api

之间有两种通信方式，它们同时使用：

kubelet

使用

--kubeconfig=/etc/kubernetes/kubelet.conf

参数中的信息连接并验证到

kube api

（您可以通过

ps-aux | grep kubelet

进行检查）。在文件中，您将看到连接字符串、CA证书和客户端证书+密钥）。Kubelet提供文件中的客户端证书，并验证CA从同一文件中获取的

kube api

服务器证书

kube api

使用在其自己的选项中定义的CA验证客户端证书

--客户端CA文件

kube-api

使用

--kubelet-client-certificate

和

--kubelet-client-key

选项连接到

kubelet

。这可能不是问题所在

因为您可以在

kubeapi

端看到SSL错误，而在

kubelet

端看不到SSL错误。我假设第n.1点中描述的通信存在问题

kubelet

连接并验证到

kube api

。错误在

kube-api

日志中，因此我认为

kube-api

在验证

kubelet

客户端证书时存在问题。所以在

--kubeconfig=/etc/kubernetes/kubelet.conf

中检查它。您可以通过openssl或一些在线SSL证书检查器对其进行base64解码并显示详细信息。最重要的部分是，它必须由

kube api

选项

--客户端CA文件中定义的CA文件签名

这一切都需要付出大量努力才能做到诚实，最简单的方法就是扔掉一切，使用
kubeadm
引导单节点集群：

清除服务器上的所有混乱


ubuntu@ubu1910-medflavor-nolb3-control-plane-nh4hf:~$ tail -F /var/log/syslog 
Apr 16 19:36:18 ubu1910-medflavor-nolb3-control-plane-nh4hf kubelet[1504]: E0416 19:36:18.596206    1504 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.RuntimeClass: Get https://172.24.4.159:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: net/http: TLS handshake timeout
Apr 16 19:36:19 ubu1910-medflavor-nolb3-control-plane-nh4hf containerd[568]: time="2021-04-16T19:36:19.202346090Z" level=error msg="Failed to load cni configuration" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Apr 16 19:36:19 ubu1910-medflavor-nolb3-control-plane-nh4hf kubelet[1504]: E0416 19:36:19.274089    1504 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Apr 16 19:36:20 ubu1910-medflavor-nolb3-control-plane-nh4hf kubelet[1504]: W0416 19:36:20.600457    1504 status_manager.go:530] Failed to get status for pod "kube-apiserver-ubu1910-medflavor-nolb3-control-plane-nh4hf_kube-system(24ec7abb1b94172adb053cf6fdd1648c)": Get https://172.24.4.159:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-ubu1910-medflavor-nolb3-control-plane-nh4hf: net/http: TLS handshake timeout
Apr 16 19:36:24 ubu1910-medflavor-nolb3-control-plane-nh4hf containerd[568]: time="2021-04-16T19:36:24.336699210Z" level=error msg="Failed to load cni configuration" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Apr 16 19:36:24 ubu1910-medflavor-nolb3-control-plane-nh4hf kubelet[1504]: E0416 19:36:24.379374    1504 controller.go:135] failed to ensure node lease exists, will retry in 7s, error: Get https://172.24.4.159:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubu1910-medflavor-nolb3-control-plane-nh4hf?timeout=10s: context deadline exceeded
......
......

ubuntu@ubu1910-medflavor-nolb3-control-plane-nh4hf:~$ curl http://172.24.4.159:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-ubu1910-medflavor-nolb3-control-plane-nh4hf
Client sent an HTTP request to an HTTPS server.

ubuntu@ubu1910-medflavor-nolb3-control-plane-nh4hf:~$ curl https://172.24.4.159:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-ubu1910-medflavor-nolb3-control-plane-nh4hf
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
ubuntu@ubu1910-medflavor-nolb3-control-plane-nh4hf:~$