Kubernetes 在GKE中的HPA中放大时出错：apiserver无法写入JSON响应：http2:stream closed

遵循google为在google Kubernetes引擎中部署HPA而制定的指南：

并添加正确的权限，因为我在本指南中使用了工作负载标识：

并在此处添加防火墙规则：

我陷入了HPA向我返回以下错误的境地：

kubectl describe hpa -n test-namespace
Name:                  my-hpa
Namespace:             test-namespace
Labels:                <none>
Annotations:           <none>
CreationTimestamp:     Tue, 13 Apr 2021 12:47:56 +0200
Reference:             StatefulSet/my-set
Metrics:               ( current / target )
  "my-metric" on pods:  <unknown> / 1
Min replicas:          1
Max replicas:          60
StatefulSet pods:      1 current / 0 desired
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric my-metric: no metrics returned from custom metrics API
Events:
  Type     Reason                        Age                   From                       Message
  ----     ------                        ----                  ----                       -------
  Warning  FailedGetPodsMetric           8m26s (x40 over 18m)  horizontal-pod-autoscaler  unable to get metric my-metric: no metrics returned from custom metrics API
  Warning  FailedComputeMetricsReplicas  3m26s (x53 over 18m)  horizontal-pod-autoscaler  failed to compute desired number of replicas based on listed metrics for StatefulSet/test-namespace/my-set: invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric my-metric: no metrics returned from custom metrics API

2021-04-13T11:01:30.432634Z apiserver was unable to write a JSON response: http2: stream closed
2021-04-13T11:01:30.432679Z apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}

当我尝试检索度量数据时，它返回ok

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta2/namespaces/test-namespace/pods/*/my-metric" | jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta2",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta2/namespaces/test-namespace/pods/%2A/my-metric"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "test-namespace",
        "name": "my-metrics-api-XXXX",
        "apiVersion": "/__internal"
      },
      "metric": {
        "name": "my-metric",
        "selector": null
      },
      "timestamp": "2021-04-13T11:15:30Z",
      "value": "5"
    }
  ]
}

但是stackdriver给了我这个错误：

kubectl describe hpa -n test-namespace
Name:                  my-hpa
Namespace:             test-namespace
Labels:                <none>
Annotations:           <none>
CreationTimestamp:     Tue, 13 Apr 2021 12:47:56 +0200
Reference:             StatefulSet/my-set
Metrics:               ( current / target )
  "my-metric" on pods:  <unknown> / 1
Min replicas:          1
Max replicas:          60
StatefulSet pods:      1 current / 0 desired
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric my-metric: no metrics returned from custom metrics API
Events:
  Type     Reason                        Age                   From                       Message
  ----     ------                        ----                  ----                       -------
  Warning  FailedGetPodsMetric           8m26s (x40 over 18m)  horizontal-pod-autoscaler  unable to get metric my-metric: no metrics returned from custom metrics API
  Warning  FailedComputeMetricsReplicas  3m26s (x53 over 18m)  horizontal-pod-autoscaler  failed to compute desired number of replicas based on listed metrics for StatefulSet/test-namespace/my-set: invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric my-metric: no metrics returned from custom metrics API

2021-04-13T11:01:30.432634Z apiserver was unable to write a JSON response: http2: stream closed
2021-04-13T11:01:30.432679Z apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}

我必须像这样配置google提供的适配器：

apiVersion: v1
kind: Namespace
metadata:
  name: custom-metrics
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: custom-metrics-stackdriver-adapter
  namespace: custom-metrics
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-metrics:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: custom-metrics-stackdriver-adapter
  namespace: custom-metrics
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: custom-metrics-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: custom-metrics-stackdriver-adapter
  namespace: custom-metrics
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: custom-metrics-resource-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
- kind: ServiceAccount
  name: custom-metrics-stackdriver-adapter
  namespace: custom-metrics
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-metrics-stackdriver-adapter
  namespace: custom-metrics
  labels:
    run: custom-metrics-stackdriver-adapter
    k8s-app: custom-metrics-stackdriver-adapter
spec:
  replicas: 1
  selector:
    matchLabels:
      run: custom-metrics-stackdriver-adapter
      k8s-app: custom-metrics-stackdriver-adapter
  template:
    metadata:
      labels:
        run: custom-metrics-stackdriver-adapter
        k8s-app: custom-metrics-stackdriver-adapter
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccountName: custom-metrics-stackdriver-adapter
      containers:
      - image: gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.0-gke.0
        imagePullPolicy: Always
        name: pod-custom-metrics-stackdriver-adapter
        command:
        - /adapter
        - --use-new-resource-model=true
        - --cert-dir=/tmp
        - --secure-port=4443
        resources:
          limits:
            cpu: 250m
            memory: 200Mi
          requests:
            cpu: 250m
            memory: 200Mi
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: custom-metrics-stackdriver-adapter
    k8s-app: custom-metrics-stackdriver-adapter
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: Adapter
  name: custom-metrics-stackdriver-adapter
  namespace: custom-metrics
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 4443
  selector:
    run: custom-metrics-stackdriver-adapter
    k8s-app: custom-metrics-stackdriver-adapter
  type: ClusterIP
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  insecureSkipTLSVerify: true
  group: custom.metrics.k8s.io
  groupPriorityMinimum: 100
  versionPriority: 100
  service:
    name: custom-metrics-stackdriver-adapter
    namespace: custom-metrics
  version: v1beta1
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta2.custom.metrics.k8s.io
spec:
  insecureSkipTLSVerify: true
  group: custom.metrics.k8s.io
  groupPriorityMinimum: 100
  versionPriority: 200
  service:
    name: custom-metrics-stackdriver-adapter
    namespace: custom-metrics
  version: v1beta2
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.external.metrics.k8s.io
spec:
  insecureSkipTLSVerify: true
  group: external.metrics.k8s.io
  groupPriorityMinimum: 100
  versionPriority: 100
  service:
    name: custom-metrics-stackdriver-adapter
    namespace: custom-metrics
  version: v1beta1
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-metrics-reader
rules:
- apiGroups:
  - "external.metrics.k8s.io"
  resources:
  - "*"
  verbs:
  - list
  - get
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-metrics-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-metrics-reader
subjects:
- kind: ServiceAccount
  name: horizontal-pod-autoscaler
  namespace: kube-system

由于端口443被禁用，因此我必须更改为4443，并放置--cert dir=/tmp选项，因为没有该选项，stackdriver会返回错误：

"unable to run custom metrics adapter: error creating self-signed certificates: mkdir apiserver.local.config: permission denied"

我想我解释了配置它的所有步骤，但没有成功。有什么想法吗？

为我解决了

经过几次试验后，HPA yaml发生变化

从

Pod

到

External

的度量，以及带有

custom.google.api/my metric

的度量名称，它可以工作

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: my-hpa
  namespace: test-namespace
spec:
  maxReplicas: 60
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: my-set
  metrics:
  - type: External
    external:
      metric: 
        name: custom.googleapis.com|my-metric
      target:
        averageValue: 1
        type: AverageValue

---

您好，您可以通过以下方式告诉我：“但是stackdriver给了我这个错误：”您是指

自定义度量stackdriver适配器的
Pod
？另外，您能告诉我您使用的是哪个
GKE
版本吗？您是否尝试过使用较旧版本的映像，如：
gcr.io/google containers/custom metrics stackdriver adapter:v0.10.1
？是的，是custom metrics stackdriver adapter。我使用的版本是1.18.16-gke.502。我没有尝试其他版本。编辑：改为
gcr.io/google containers/custom metrics stackdriver adapter:v0.10.1
version stackdriver，编写JSON时不再出现错误，但HPA中的问题是sameI注意到执行
kubectl get--raw”/api/custom.metrics.k8s.io/v1beta1| jq.
resources

变量是一个空数组，如果这有帮助的话