Docker filebeat+；库伯内特斯+；elasticsearch不保存特定字段

我创建了一个名称空间来获取带有filebeats的日志并保存到elasticsearch。 为什么不在elasticsearch上保存关于Kubernetes的字段如何遵循示例

当前正在按如下方式保存：

      "_source" : {
          "@timestamp" : "2020-01-23T12:33:14.235Z",
          "ecs" : {
            "version" : "1.0.0"
          },
          "host" : {
            "name" : "worker-node1"
          },
          "agent" : {
            "hostname" : "worker-node1",
            "id" : "xxxxx-xxxx-xxx-xxxx-xxxxxxxxxxxxxx",
            "version" : "7.1.1",
            "type" : "filebeat",
            "ephemeral_id" : "xxxx-xxxx-xxxx-xxxxxxxxxxxxx"
          },
          "log" : {
            "offset" : xxxxxxxx,
            "file" : {
              "path" : "/var/lib/docker/containers/xxxx96ec2bfd9a3e4f4ac83581ad90/7fd55e1249aa009df3f8e3250c967bbe541c9596xxxxxac83581ad90-json.log"
            }
          },
          "stream" : "stdout",
          "message" : "xxxxxxxx",
          "input" : {
            "type" : "docker"
          }
        }

要遵循我的filebeat.config，请执行以下操作：

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.config:
      inputs:
        # Mounted `filebeat-inputs` configmap:
        path: ${path.config}/inputs.d/*.yml
        # Reload inputs configs as they change:
        reload.enabled: false
        multiline.pattern: '^[[:space:]]'
        multiline.negate: false
        multiline.match: after
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false

    # To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      hints.enabled: true

    processors:
      - add_cloud_metadata:
      - add_kubernetes_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      protocol: "http"
    setup.ilm.enabled: false
    ilm.enabled: false
    xpack.monitoring:
      enabled: true

DamemonSet如下所示：

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      hostNetwork: true
      terminationGracePeriodSeconds: 30
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat-oss:7.1.1
          args: [
            "-c", "/etc/filebeat.yml",
            "-e",
          ]
          env:
            - name: ELASTICSEARCH_HOST
              value: xxxxxxxxxxxxx
            - name: ELASTICSEARCH_PORT
              value: "9200"
          securityContext:
            runAsUser: 0
            # If using Red Hat OpenShift uncomment this:
            #privileged: true
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat.yml
              readOnly: true
              subPath: filebeat.yml
            - name: inputs
              mountPath: /usr/share/filebeat/inputs.d
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config
          configMap:
            defaultMode: 0600
            name: filebeat-config
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: inputs
          configMap:
            defaultMode: 0600
            name: filebeat-inputs
        # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate

---

在将配置应用于kubernetes之前，我确实删除了elasticsearch的任何注册表文件节拍。

正如我在评论中所述。您的

ConfigMap

似乎缺少容器日志的

路径：

。应该是这样的：

       type: container
       paths:
         - /var/log/containers/*${data.kubernetes.container.id}.log

将您的配置文件与

---

我希望它能有所帮助。

我也遇到了同样的问题，我通过从守护程序中删除hostNetwork:true配置来解决。这意味着pod名称与节点名称相同。查看filebeat启动日志，您可以看到这一点。

请编辑您的问题，更好地解释您所做的工作以及您希望实现的目标。您的问题到底是什么？我正在使用filebeat在elasticsearch上保存字段，我需要保存关于我的kubernetes的字段。看起来您的ConfigMap缺少容器日志的路径：。类似于：

/var/log/containers/*${data.kubernetes.container.id}.log

。请将您的配置与进行比较。您是否设法使其正常工作？

       type: container
       paths:
         - /var/log/containers/*${data.kubernetes.container.id}.log