用于spring引导日志的Logstash筛选器，用于捕获日志级别、类名

我正在尝试使用下面的配置在过滤器块中使用logstash来过滤SpringBoot应用程序日志

filter {

 grok {
    match => [ "message",
               "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- \[(?<thread>[A-Za-z0-9-]+)\] [A-Za-z0-9.]*\.(?<class>[A-Za-z0-9#_]+)\s*:\s+(?<logmessage>.*)",
               "message",
               "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- .+? :\s+(?<logmessage>.*)"
             ]
  }


  date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
  }
}

我应该在grok筛选器中更改什么以捕获日志级别、类名等。

---

提前感谢。

由于您的消息跨越多行（与stacktraces通常一样），因此您需要在regex/grok模式中添加多行标志m（参见此示例）

解析stacktraces不是一项容易的任务。因此，我并不期望通过添加多行标志，它会自动工作。我想你需要经常调试它

但我认为您确实需要使用多行标志。

尝试以下方法：

输入：

2020-01-23 12:08:51.468 ERROR 13216 --- [http-nio-8085-exec-1] com.poc.SampleLog.DemoController         : java.lang.NullPointerException

java.lang.NullPointerException: null
    at com.poc.SampleLog.DemoController.exception2(DemoController.java:36) ~[classes/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]

(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:level} %{NUMBER:pid} --- \[%{GREEDYDATA:thread}\] %{GREEDYDATA:class}: %{GREEDYDATA:exception}%{SPACE}(?<stacktrace>(.|\r|\n)*)

{
  "timestamp": [
    [
      "2020-01-23 12:08:51.468"
    ]
  ],
  "YEAR": [
    [
      "2020"
    ]
  ],
  "MONTHNUM": [
    [
      "01"
    ]
  ],
  "MONTHDAY": [
    [
      "23"
    ]
  ],
  "TIME": [
    [
      "12:08:51.468"
    ]
  ],
  "HOUR": [
    [
      "12"
    ]
  ],
  "MINUTE": [
    [
      "08"
    ]
  ],
  "SECOND": [
    [
      "51.468"
    ]
  ],
  "level": [
    [
      "ERROR"
    ]
  ],
  "pid": [
    [
      "13216"
    ]
  ],
  "BASE10NUM": [
    [
      "13216"
    ]
  ],
  "thread": [
    [
      "http-nio-8085-exec-1"
    ]
  ],
  "class": [
    [
      "com.poc.SampleLog.DemoController         "
    ]
  ],
  "exception": [
    [
      "java.lang.NullPointerException"
    ]
  ],
  "SPACE": [
    [
      "\n\n"
    ]
  ],
  "stacktrace": [
    [
      "java.lang.NullPointerException: null\n    at com.poc.SampleLog.DemoController.exception2(DemoController.java:36) ~[classes/:na]\n    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]\n    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]\n    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]\n    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]"
    ]
  ]
}

GROK模式：

2020-01-23 12:08:51.468 ERROR 13216 --- [http-nio-8085-exec-1] com.poc.SampleLog.DemoController         : java.lang.NullPointerException

java.lang.NullPointerException: null
    at com.poc.SampleLog.DemoController.exception2(DemoController.java:36) ~[classes/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]

(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:level} %{NUMBER:pid} --- \[%{GREEDYDATA:thread}\] %{GREEDYDATA:class}: %{GREEDYDATA:exception}%{SPACE}(?<stacktrace>(.|\r|\n)*)

{
  "timestamp": [
    [
      "2020-01-23 12:08:51.468"
    ]
  ],
  "YEAR": [
    [
      "2020"
    ]
  ],
  "MONTHNUM": [
    [
      "01"
    ]
  ],
  "MONTHDAY": [
    [
      "23"
    ]
  ],
  "TIME": [
    [
      "12:08:51.468"
    ]
  ],
  "HOUR": [
    [
      "12"
    ]
  ],
  "MINUTE": [
    [
      "08"
    ]
  ],
  "SECOND": [
    [
      "51.468"
    ]
  ],
  "level": [
    [
      "ERROR"
    ]
  ],
  "pid": [
    [
      "13216"
    ]
  ],
  "BASE10NUM": [
    [
      "13216"
    ]
  ],
  "thread": [
    [
      "http-nio-8085-exec-1"
    ]
  ],
  "class": [
    [
      "com.poc.SampleLog.DemoController         "
    ]
  ],
  "exception": [
    [
      "java.lang.NullPointerException"
    ]
  ],
  "SPACE": [
    [
      "\n\n"
    ]
  ],
  "stacktrace": [
    [
      "java.lang.NullPointerException: null\n    at com.poc.SampleLog.DemoController.exception2(DemoController.java:36) ~[classes/:na]\n    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]\n    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]\n    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]\n    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]"
    ]
  ]
}

---

这种模式适用于默认的spring.log文件

filter {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:time}\s+%{LOGLEVEL:log_level}\s+\[%{DATA:appName},%{DATA:traceId},%{DATA:spanId},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:messageTmp}" }
#      remove_field => "message"
    }

    date {
      match => ["time", "YYYY-MM-dd HH:mm:ss.SSS"]
      target => "@timestamp"
      remove_field => "time"
    }

    mutate {
      add_field => {"serviceName" => "back"}
      rename => {"messageTmp" => "message"}
    }
}

---

您可以添加一些示例日志行吗？也许可以尝试在中重新加载索引模式Kibana@IsharaMadhawa：我已添加示例日志lines@baudsp：我尝试了新的索引模式也没有帮助谢谢你的答案。现在它正在捕获日志级别，但不是正确的类。它如下所示：类com.poc.SampleLog.DemoController:java.lang.arithmetricexception:/by zero java.lang.arithmetricexceptionexception/by zero位于com.poc.SampleLog.DemoController.exception1（DemoController.java:28）~[classes/:na]位于java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0（N。。。