Fluentd解析部分json日志

我有以下来自ApacheDruid的日志

{"timeMillis":1587714600921,"thread":"qtp150208546-149","level":"INFO","loggerName":"org.apache.druid.server.log.LoggingRequestLogger","message":"2020-04-24T07:50:00.798Z\t10.2.73.23\t{\"queryType\":\"segmentMetadata\",\"dataSource\":{\"type\":\"table\",\"name\":\"tableB\"},\"intervals\":{\"type\":\"LegacySegmentSpec\",\"intervals\":[\"1999-01-01T00:00:00.000Z/2114-01-01T00:00:00.000Z\"]},\"toInclude\":{\"type\":\"all\"},\"merge\":true,\"context\":{\"queryId\":\"ed1e7129-1e3f-438d-acbb-04f11d292eb5\"},\"analysisTypes\":[\"aggregators\"],\"usingDefaultInterval\":false,\"lenientAggregatorMerge\":false,\"descending\":false,\"granularity\":{\"type\":\"all\"}}\t{\"query/time\":122,\"query/bytes\":4712,\"success\":true,\"identity\":\"allowAll\"}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger"}
{"timeMillis":1587714600952,"thread":"qtp150208546-119","level":"INFO","loggerName":"org.apache.druid.server.log.LoggingRequestLogger","message":"2020-04-24T07:50:00.941Z\t10.2.73.23\t{\"queryType\":\"segmentMetadata\",\"dataSource\":{\"type\":\"table\",\"name\":\"test\"},\"intervals\":{\"type\":\"LegacySegmentSpec\",\"intervals\":[\"1999-01-01T00:00:00.000Z/2114-01-01T00:00:00.000Z\"]},\"toInclude\":{\"type\":\"all\"},\"merge\":true,\"context\":{\"queryId\":\"6368e8c4-e18d-4a29-97cc-df2e0aadd02e\"},\"analysisTypes\":[\"aggregators\"],\"usingDefaultInterval\":false,\"lenientAggregatorMerge\":false,\"descending\":false,\"granularity\":{\"type\":\"all\"}}\t{\"query/time\":10,\"query/bytes\":4710,\"success\":true,\"identity\":\"allowAll\"}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger"}
{"timeMillis":1587714662763,"thread":"qtp150208546-131","level":"INFO","loggerName":"org.apache.druid.server.log.LoggingRequestLogger","message":"2020-04-24T07:51:02.694Z\t\t{\"queryType\":\"topN\",\"dataSource\":{\"type\":\"table\",\"name\":\"tableA\"},\"virtualColumns\":[],\"dimension\":{\"type\":\"default\",\"dimension\":\"key\",\"outputName\":\"d0\",\"outputType\":\"STRING\"},\"metric\":{\"type\":\"numeric\",\"metric\":\"a0\"},\"threshold\":100,\"intervals\":{\"type\":\"intervals\",\"intervals\":[\"2020-03-05T07:51:02.000Z/146140482-04-24T15:36:27.903Z\"]},\"filter\":null,\"granularity\":{\"type\":\"all\"},\"aggregations\":[{\"type\":\"count\",\"name\":\"a0\"}],\"postAggregations\":[],\"context\":{\"queryId\":\"2084372b-f9ec-43c4-a5c7-bb6c28a738fc\",\"sqlQueryId\":\"e137d0ef-38ba-4af5-ac0e-5cdf86439ace\"},\"descending\":false}\t{\"query/time\":68,\"query/bytes\":-1,\"success\":true,\"identity\":\"allowAll\"}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger"}
{"timeMillis":1587714662763,"thread":"qtp150208546-131","level":"INFO","loggerName":"org.apache.druid.server.log.LoggingRequestLogger","message":"2020-04-24T07:51:02.642Z\t10.2.64.24\t\t{\"sqlQuery/time\":121,\"sqlQuery/bytes\":2095,\"success\":true,\"context\":{\"sqlQueryId\":\"e137d0ef-38ba-4af5-ac0e-5cdf86439ace\",\"nativeQueryIds\":\"[2084372b-f9ec-43c4-a5c7-bb6c28a738fc]\"},\"identity\":\"allowAll\"}\t{\"query\":\"SELECT * FROM (SELECT\\n  \\\"key\\\",\\n  COUNT(*) AS \\\"Count\\\"\\nFROM \\\"tableA\\\"\\nWHERE \\\"__time\\\" >= CURRENT_TIMESTAMP - INTERVAL '50' DAY\\nGROUP BY 1\\nORDER BY \\\"Count\\\" DESC\\n) LIMIT 100\",\"context\":{\"sqlQueryId\":\"e137d0ef-38ba-4af5-ac0e-5cdf86439ace\",\"nativeQueryIds\":\"[2084372b-f9ec-43c4-a5c7-bb6c28a738fc]\"}}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger"}

我想使用JSON解析器解析键

message

，以获取每个提取的字段和值

"2020-04-24T07:50:00.798Z\t10.2.73.23\t{\"queryType\":\"segmentMetadata\",\"dataSource\":{\"type\":\"table\",\"name\":\"tableA\"},\"intervals\":{\"type\":\"LegacySegmentSpec\",\"intervals\":[\"1999-01-01T00:00:00.000Z/2114-01-01T00:00:00.000Z\"]},\"toInclude\":{\"type\":\"all\"},\"merge\":true,\"context\":{\"queryId\":\"ed1e7129-1e3f-438d-acbb-04f11d292eb5\"},\"analysisTypes\":[\"aggregators\"],\"usingDefaultInterval\":false,\"lenientAggregatorMerge\":false,\"descending\":false,\"granularity\":{\"type\":\"all\"}}\t{\"query/time\":122,\"query/bytes\":4712,\"success\":true,\"identity\":\"allowAll\"}"
"2020-04-24T07:50:00.941Z\t10.2.73.23\t{\"queryType\":\"segmentMetadata\",\"dataSource\":{\"type\":\"table\",\"name\":\"test\"},\"intervals\":{\"type\":\"LegacySegmentSpec\",\"intervals\":[\"1999-01-01T00:00:00.000Z/2114-01-01T00:00:00.000Z\"]},\"toInclude\":{\"type\":\"all\"},\"merge\":true,\"context\":{\"queryId\":\"6368e8c4-e18d-4a29-97cc-df2e0aadd02e\"},\"analysisTypes\":[\"aggregators\"],\"usingDefaultInterval\":false,\"lenientAggregatorMerge\":false,\"descending\":false,\"granularity\":{\"type\":\"all\"}}\t{\"query/time\":10,\"query/bytes\":4710,\"success\":true,\"identity\":\"allowAll\"}"
"2020-04-24T07:51:02.694Z\t\t{\"queryType\":\"topN\",\"dataSource\":{\"type\":\"table\",\"name\":\"tableA\"},\"virtualColumns\":[],\"dimension\":{\"type\":\"default\",\"dimension\":\"key\",\"outputName\":\"d0\",\"outputType\":\"STRING\"},\"metric\":{\"type\":\"numeric\",\"metric\":\"a0\"},\"threshold\":100,\"intervals\":{\"type\":\"intervals\",\"intervals\":[\"2020-03-05T07:51:02.000Z/146140482-04-24T15:36:27.903Z\"]},\"filter\":null,\"granularity\":{\"type\":\"all\"},\"aggregations\":[{\"type\":\"count\",\"name\":\"a0\"}],\"postAggregations\":[],\"context\":{\"queryId\":\"2084372b-f9ec-43c4-a5c7-bb6c28a738fc\",\"sqlQueryId\":\"e137d0ef-38ba-4af5-ac0e-5cdf86439ace\"},\"descending\":false}\t{\"query/time\":68,\"query/bytes\":-1,\"success\":true,\"identity\":\"allowAll\"}"
"2020-04-24T07:51:02.642Z\t10.2.64.24\t\t{\"sqlQuery/time\":121,\"sqlQuery/bytes\":2095,\"success\":true,\"context\":{\"sqlQueryId\":\"e137d0ef-38ba-4af5-ac0e-5cdf86439ace\",\"nativeQueryIds\":\"[2084372b-f9ec-43c4-a5c7-bb6c28a738fc]\"},\"identity\":\"allowAll\"}\t{\"query\":\"SELECT * FROM (SELECT\\n  \\\"key\\\",\\n  COUNT(*) AS \\\"Count\\\"\\nFROM \\\"tableA\\\"\\nWHERE \\\"__time\\\" >= CURRENT_TIMESTAMP - INTERVAL '50' DAY\\nGROUP BY 1\\nORDER BY \\\"Count\\\" DESC\\n) LIMIT 100\",\"context\":{\"sqlQueryId\":\"e137d0ef-38ba-4af5-ac0e-5cdf86439ace\",\"nativeQueryIds\":\"[2084372b-f9ec-43c4-a5c7-bb6c28a738fc]\"}}"

正如您可以看到的，它不是JSON格式。但我只想提取JSON部分。
Fluentd版本：

v1.8.1

我正在尝试此Fluentd配置：

<filter kubernetes.var.log.containers.druid-brokers-**.log>
      @type parser
      key_name $["log"]["message"]
      reserve_data true
      remove_key_name_field true
      hash_value_field parsed
      <parse>
        @type json
      </parse>
</filter>

谢谢你的帮助

问候,，

---

Vincent

我认为

消息

字段必须访问为：

key_name $["message"]

---

我也遇到了同样的错误，结果是我试图访问的日志记录中的元素不正确，即路径不存在。

我认为必须访问

消息

字段，如下所示：

key_name $["message"]

我也遇到了同样的错误，结果是我试图访问的日志记录中的元素不正确，即路径不存在