Kubernetes kube apiserver启动时无法获取或设置密钥

我正在尝试使用packer和terraform而不是kube-up.sh脚本设置一个高可用的kubernetes集群。原因：我想要更大的机器，不同的设置等等。我的大部分配置来自coreos kubernetes部署教程

关于我的设置：

科雷奥斯

一切都在gce上运行。 我有3个etcd和一个skydns实例。他们正在工作，能够相互联系

我有一个实例作为kubernetes主实例，它运行带有清单的kubelet

我现在的实际问题是kube api服务器本身无法连接到它。我可以从主机系统运行curl命令，并获得有效响应/版本和其他

还有一点奇怪，443和8080不是从docker转发过来的。或者这是正常的行为

我想我配置错了一些主端点。因此，我尝试了本地主机和外部ip的所有清单。=>不工作

kube api容器中的错误：

I0925 14:51:47.505859       1 plugins.go:69] No cloud provider specified.
I0925 14:51:47.973450       1 master.go:273] Node port range unspecified. Defaulting to 30000-32767.
E0925 14:51:48.009367       1 reflector.go:136] Failed to list *api.ResourceQuota: Get http://127.0.0.1:8080/api/v1/resourcequotas: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.010730       1 reflector.go:136] Failed to list *api.Secret: Get http://127.0.0.1:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.010996       1 reflector.go:136] Failed to list *api.ServiceAccount: Get http://127.0.0.1:8080/api/v1/serviceaccounts: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.011083       1 reflector.go:136] Failed to list *api.LimitRange: Get http://127.0.0.1:8080/api/v1/limitranges: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.012697       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.012753       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
[restful] 2015/09/25 14:51:48 log.go:30: [restful/swagger] listing is available at https://104.155.60.74:443/swaggerapi/
[restful] 2015/09/25 14:51:48 log.go:30: [restful/swagger] https://104.155.60.74:443/swaggerui/ is mapped to folder /swagger-ui/
I0925 14:51:48.136166       1 server.go:441] Serving securely on 0.0.0.0:443
I0925 14:51:48.136248       1 server.go:483] Serving insecurely on 127.0.0.1:8080

控制器容器具有几乎相同的错误。其他容器都可以

我的配置：

I0925 14:51:47.505859       1 plugins.go:69] No cloud provider specified.
I0925 14:51:47.973450       1 master.go:273] Node port range unspecified. Defaulting to 30000-32767.
E0925 14:51:48.009367       1 reflector.go:136] Failed to list *api.ResourceQuota: Get http://127.0.0.1:8080/api/v1/resourcequotas: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.010730       1 reflector.go:136] Failed to list *api.Secret: Get http://127.0.0.1:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.010996       1 reflector.go:136] Failed to list *api.ServiceAccount: Get http://127.0.0.1:8080/api/v1/serviceaccounts: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.011083       1 reflector.go:136] Failed to list *api.LimitRange: Get http://127.0.0.1:8080/api/v1/limitranges: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.012697       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
E0925 14:51:48.012753       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
[restful] 2015/09/25 14:51:48 log.go:30: [restful/swagger] listing is available at https://104.155.60.74:443/swaggerapi/
[restful] 2015/09/25 14:51:48 log.go:30: [restful/swagger] https://104.155.60.74:443/swaggerui/ is mapped to folder /swagger-ui/
I0925 14:51:48.136166       1 server.go:441] Serving securely on 0.0.0.0:443
I0925 14:51:48.136248       1 server.go:483] Serving insecurely on 127.0.0.1:8080

/etc/kubelet.env

KUBE_KUBELET_OPTS="\
  --api_servers=http://127.0.0.1:8080 \
  --register-node=false \
  --allow-privileged=true \
  --config=/etc/kubernetes/manifests \
  --tls_cert_file=/etc/kubernetes/ssl/apiserver.pem \
  --tls_private_key_file=/etc/kubernetes/ssl/apiserver-key.pem \
  --cloud-provider=gce \
  --cluster_dns=10.10.38.10 \
  --cluster_domain=cluster.local \
  --cadvisor-port=0"

/etc/kubernetes/manifests/

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: kube-apiserver
    image: gcr.io/google_containers/hyperkube:v1.0.6
    command:
    - /hyperkube
    - apiserver
    - --bind-address=0.0.0.0
    - --etcd_servers=http://10.10.125.10:2379,http://10.10.82.201:2379,http://10.10.63.185:2379
    - --allow-privileged=true
    - --service-cluster-ip-range=10.40.0.0/16
    - --secure_port=443
    - --advertise-address=104.155.60.74
    - --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota
    - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
    - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    - --client-ca-file=/etc/kubernetes/ssl/ca.pem
    - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    ports:
    - containerPort: 443
      hostPort: 443
      name: https
    - containerPort: 8080
      hostPort: 8080
      name: local
    volumeMounts:
    - mountPath: /etc/kubernetes/ssl
      name: ssl-certs-kubernetes
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ssl-certs-host
      readOnly: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/ssl
    name: ssl-certs-kubernetes
  - hostPath:
      path: /usr/share/ca-certificates
    name: ssl-certs-host

/etc/kubernetes/manifests/kube-controller-manager.yml

apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - name: kube-controller-manager
    image: gcr.io/google_containers/hyperkube:v1.0.6
    command:
    - /hyperkube
    - controller-manager
    - --master=https://104.155.60.74:443
    - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    - --root-ca-file=/etc/kubernetes/ssl/ca.pem
    - --cloud_provider=gce
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10252
      initialDelaySeconds: 15
      timeoutSeconds: 1
    volumeMounts:
    - mountPath: /etc/kubernetes/ssl
      name: ssl-certs-kubernetes
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ssl-certs-host
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/ssl
    name: ssl-certs-kubernetes
  - hostPath:
      path: /usr/share/ca-certificates
    name: ssl-certs-host

docker ps

CONTAINER ID        IMAGE                                       COMMAND                CREATED             STATUS              PORTS               NAMES
3e37b2ea2277        gcr.io/google_containers/hyperkube:v1.0.6   "/hyperkube controll   31 minutes ago      Up 31 minutes                           k8s_kube-controller-manager.afecd3c9_kube-controller-manager-kubernetes-km0.c.stylelounge-1042.inte
rnal_kube-system_621db46bf7b0764eaa46d17dfba8e90f_519cd0da   
43917185d91b        gcr.io/google_containers/hyperkube:v1.0.6   "/hyperkube proxy --   31 minutes ago      Up 31 minutes                           k8s_kube-proxy.a2db3197_kube-proxy-kubernetes-km0.c.stylelounge-1042.internal_kube-system_67c22e99a
eb1ef9c2997c942cfbe48b9_c82a8a60                             
f548279e90f9        gcr.io/google_containers/hyperkube:v1.0.6   "/hyperkube apiserve   31 minutes ago      Up 31 minutes                           k8s_kube-apiserver.2bcb2c35_kube-apiserver-kubernetes-km0.c.stylelounge-1042.internal_kube-system_8
67c500deb54965609810fd0771fa92d_a306feae                     
94b1942a09f0        gcr.io/google_containers/hyperkube:v1.0.6   "/hyperkube schedule   31 minutes ago      Up 31 minutes                           k8s_kube-scheduler.603b59f4_kube-scheduler-kubernetes-km0.c.stylelounge-1042.internal_kube-system_3
9e2c582fd067b44ebe8cefaee036c0e_e0ddf6a2                     
9de4a4264ef6        gcr.io/google_containers/podmaster:1.1      "/podmaster --etcd-s   31 minutes ago      Up 31 minutes                           k8s_controller-manager-elector.89f472b4_kube-podmaster-kubernetes-km0.c.stylelounge-1042.internal_k
ube-system_e23fc0902c7e6da7b315ad34130b9807_7c8d2901         
af2df45f4081        gcr.io/google_containers/podmaster:1.1      "/podmaster --etcd-s   31 minutes ago      Up 31 minutes                           k8s_scheduler-elector.608b6780_kube-podmaster-kubernetes-km0.c.stylelounge-1042.internal_kube-syste
m_e23fc0902c7e6da7b315ad34130b9807_b11e601d                  
ac0e068456c7        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-controller-manager-kubernetes-km0.c.stylelounge-1042.internal_kube-system_621d
b46bf7b0764eaa46d17dfba8e90f_e9760e28                        
2773ba48d011        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-podmaster-kubernetes-km0.c.stylelounge-1042.internal_kube-system_e23fc0902c7e6
da7b315ad34130b9807_4fba9edb                                 
987531f1951d        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-apiserver-kubernetes-km0.c.stylelounge-1042.internal_kube-system_867c500deb549
65609810fd0771fa92d_d15d2d66                                 
f4453b948186        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-proxy-kubernetes-km0.c.stylelounge-1042.internal_kube-system_67c22e99aeb1ef9c2
997c942cfbe48b9_07e540c8                                     
ce01cfda007e        gcr.io/google_containers/pause:0.8.0        "/pause"               31 minutes ago      Up 31 minutes                           k8s_POD.e4cc795_kube-scheduler-kubernetes-km0.c.stylelounge-1042.internal_kube-system_39e2c582fd067
b44ebe8cefaee036c0e_e6cb6500          

下面是curl命令：

kubernetes-km0 ~ # docker logs a404a310b55e
I0928 09:14:05.019135       1 plugins.go:69] No cloud provider specified.
I0928 09:14:05.192451       1 master.go:273] Node port range unspecified. Defaulting to 30000-32767.
I0928 09:14:05.192900       1 master.go:295] Will report 10.10.247.127 as public IP address.
E0928 09:14:05.226222       1 reflector.go:136] Failed to list *api.LimitRange: Get http://127.0.0.1:8080/api/v1/limitranges: dial tcp 127.0.0.1:8080: connection refused
E0928 09:14:05.226428       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
E0928 09:14:05.226479       1 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused
E0928 09:14:05.226593       1 reflector.go:136] Failed to list *api.Secret: Get http://127.0.0.1:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token: dial tcp 127.0.0.1:8080: connection refused
E0928 09:14:05.226908       1 reflector.go:136] Failed to list *api.ServiceAccount: Get http://127.0.0.1:8080/api/v1/serviceaccounts: dial tcp 127.0.0.1:8080: connection refused
[restful] 2015/09/28 09:14:05 log.go:30: [restful/swagger] listing is available at https://10.10.247.127:443/swaggerapi/
[restful] 2015/09/28 09:14:05 log.go:30: [restful/swagger] https://10.10.247.127:443/swaggerui/ is mapped to folder /swagger-ui/
E0928 09:14:05.232632       1 reflector.go:136] Failed to list *api.ResourceQuota: Get http://127.0.0.1:8080/api/v1/resourcequotas: dial tcp 127.0.0.1:8080: connection refused
I0928 09:14:05.368697       1 server.go:441] Serving securely on 0.0.0.0:443
I0928 09:14:05.368788       1 server.go:483] Serving insecurely on 127.0.0.1:8080
kubernetes-km0 ~ # curl http://127.0.0.1:8080/api/v1/limitranges
{
  "kind": "LimitRangeList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/limitranges",
    "resourceVersion": "100"
  },
  "items": []
}

---

---

我认为您需要指定--unsecure address=127.0.0.1和--unsecure port=8080才能在HTTP上打开，默认值是https。

如果您希望主机实际承载任何带有kubelet runnning on master的

--register node=true

标志的POD，则需要将主机注册为节点。CoreOs教程没有将主机注册为节点，因为这是理想情况。

I添加了不安全的绑定地址和不安全的端口，但默认值为127.0.0.1和8080。未更改任何内容。--不安全的绑定地址和端口具有默认值。改变它不会有什么不同。