Logstash 在Linux（Ubuntu）中提取系统日志的Grok模式

如果您有任何grok模式可以在ubuntu中提取系统日志，请提供它。谢谢大家!

编辑--->>

我的系统日志示例->

"Aug 20 15:53:02 amantha-ubuntu-server kibana[1877]: {\"type\":\"response\",\"@timestamp\":\"2020-08-20T10:23:02Z\",\"tags\":[],\"pid\":1877,\"method\":\"post\",\"statusCode\":200,\"req\":{\"url\":\"/internal/search/es\",\"method\":\"post\",\"headers\":{\"connection\":\"upgrade\",\"host\":\"example.com\",\"content-length\":\"861\",\"kbn-version\":\"7.8.1\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36\",\"content-type\":\"application/json\",\"accept\":\"*/*\",\"origin\":\"http://example.com\",\"referer\":\"http://example.com/app/kibana\",\"accept-encoding\":\"gzip, deflate\",\"accept-language\":\"en-US,en;q=0.9,si;q=0.8\"},\"remoteAddress\":\"127.0.0.1\",\"userAgent\":\"127.0.0.1\",\"referer\":\"http://example.com/app/kibana\"},\"res\":{\"statusCode\":200,\"responseTime\":65,\"contentLength\":9},\"message\":\"POST /internal/search/es 200 65ms - 9.0B\"}"

filter {
      grok {
            match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"}
            #match => {"syslog_message" => "%{WORD:FILTERED}" }
            #add_field => [ "received_at", "%{@timestamp}" ]
            #add_field => [ "received_from", "%{host}" ]
            remove_field => ["host","message"]
            
      }

      mutate{

            rename => ["@timestamp","time"]

      }
}

我尝试了以下过滤器->

"Aug 20 15:53:02 amantha-ubuntu-server kibana[1877]: {\"type\":\"response\",\"@timestamp\":\"2020-08-20T10:23:02Z\",\"tags\":[],\"pid\":1877,\"method\":\"post\",\"statusCode\":200,\"req\":{\"url\":\"/internal/search/es\",\"method\":\"post\",\"headers\":{\"connection\":\"upgrade\",\"host\":\"example.com\",\"content-length\":\"861\",\"kbn-version\":\"7.8.1\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36\",\"content-type\":\"application/json\",\"accept\":\"*/*\",\"origin\":\"http://example.com\",\"referer\":\"http://example.com/app/kibana\",\"accept-encoding\":\"gzip, deflate\",\"accept-language\":\"en-US,en;q=0.9,si;q=0.8\"},\"remoteAddress\":\"127.0.0.1\",\"userAgent\":\"127.0.0.1\",\"referer\":\"http://example.com/app/kibana\"},\"res\":{\"statusCode\":200,\"responseTime\":65,\"contentLength\":9},\"message\":\"POST /internal/search/es 200 65ms - 9.0B\"}"

filter {
      grok {
            match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"}
            #match => {"syslog_message" => "%{WORD:FILTERED}" }
            #add_field => [ "received_at", "%{@timestamp}" ]
            #add_field => [ "received_from", "%{host}" ]
            remove_field => ["host","message"]
            
      }

      mutate{

            rename => ["@timestamp","time"]

      }
}

然后我得到了下面的输出。我想提取系统日志消息部分。

 "time" => 2020-08-20T11:17:57.995Z,
      "syslog_message" => "message repeated 9 times: [ {\"type\":\"response\",\"@timestamp\":\"2020-08-20T10:27:22Z\",\"tags\":[],\"pid\":1877,\"method\":\"get\",\"statusCode\":200,\"req\":{\"url\":\"/api/rollup/indices\",\"method\":\"get\",\"headers\":{\"connection\":\"upgrade\",\"host\":\"example.com\",\"kbn-version\":\"7.8.1\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36\",\"content-type\":\"application/json\",\"accept\":\"*/*\",\"referer\":\"http://example.com/app/kibana\",\"accept-encoding\":\"gzip, deflate\",\"accept-language\":\"en-US,en;q=0.9,si;q=0.8\"},\"remoteAddress\":\"127.0.0.1\",\"userAgent\":\"127.0.0.1\",\"referer\":\"http://example.com/app/kibana\"},\"res\":{\"statusCode\":200,\"responseTime\":31,\"contentLength\":9},\"message\":\"GET /api/rollup/indices 200 31ms - 9.0B\"},]",
    "syslog_timestamp" => "Aug 20 15:58:52",
                "path" => "/var/log/syslog",
            "@version" => "1",
     "syslog_hostname" => "amantha-ubuntu-server",
      "syslog_program" => "amantha"

---

我要做的第一件事是在那一行中删除

“

和

\

您可以使用：

这将给您留下：

---

Aug 20 15:53:02 amantha ubuntu服务器kibana[1877]：{type:response，@timestamp:2020-08-20T10:23:02Z，标签：[]，pid:1877，方法：post，状态码：200，请求：{url:/internal/search/es，方法：post，标题：{connection:upgrade，host:example.com，内容长度：861，kbn版本：7.8.1，用户代理：Mozilla/5.0（X11；Linux x8664）AppleWebKit/537.36（KHTML，比如Gecko）Chrome/84.0.4147.125 Safari/537.36，内容类型：application/json，accept://*，来源：http://example.com，请参阅：http://example.com/app/kibana，接受编码：gzip，deflate，接受语言：en-US，en；q=0.9，si；q=0.8}，远程地址：127.0.0.1，用户代理：127.0.0.1，参考：http://example.com/app/kibana}，res:{statusCode:200，responseTime:65，contentLength:9}，message:POST/internal/search/es 200 65ms-9.0B}

以您的系统日志中的一些行作为示例添加到问题中！非常感谢。我想中断消息的其余部分以获取浏览集类型、客户端ip等。您能帮我吗？您能提供一个过滤器来中断整个系统日志吗？是的，我会尽快尝试。现在需要崩溃，但我会尽快回复这是可能的。非常感谢。我因为这个问题而感到困惑。如果你提供了过滤器，这将是一个很大的帮助。好的，给你：这应该可以解决问题。如果这对你有帮助，请让我知道。我会通知你的。谢谢。给我一些时间