Regex 日志存储解析日志字段

我试图从后缀日志中解析

@message

字段，并将其提取到多个字段中

消息：

<22>Sep 17 19:12:14 postfix/smtp[18852]: 28D40A036B: to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9030A15D0)

{
  "@source": "syslog://192.244.100.42/",
  "@tags": [
    "_grokparsefailure"
  ],
  "@fields": {
    "priority": 13,
    "severity": 5,
    "facility": 1,
    "facility_label": "user-level",
    "severity_label": "Notice"
  },
  "@timestamp": "2013-09-17T17:12:06.958Z",
  "@source_host": "192.244.100.42",
  "@source_path": "/",
  "@message": "<22>Sep 17 19:12:14 postfix/smtp[18852]: 28D40A036B: to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9030A15D0)",
  "@type": "syslog"
}

filter {
    if [type] == "postfix" {
        grok {
            patterns_dir   => "/etc/logstash/patterns"
            match => { "message" => "%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} %{SYSLOGPROG}: %{POSTFIX_SMTP_DELIVERY}" }
        }
    }
}

{
    "postfix_queueid" => "28D40A036B",
    "@timestamp" => 2017-02-23T08:15:32.546Z,
    "postfix_smtp_response" => "250 2.0.0 Ok: queued as 9030A15D0",
    "port" => 50228,
    "postfix_keyvalue_data" => "to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent",
    "syslog5424_pri" => "22",
    "@version" => "1",
    "host" => "10.0.2.2",
    "pid" => "18852",
    "program" => "postfix/smtp",
    "message" => "<22>Sep 17 19:12:14 postfix/smtp[18852]: 28D40A036B: to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9030A15D0)"
}

Sep 17 19:12:14 postfix/smtp[18852]：28D40A036B:to=，relay=192.244.100.25[192.244.100.25]：25，delay=0.13，delays=0.01/0.01/0.09/0.02，dsn=2.0.0，status=sent（250 2.0.0.0确定：排队为9030A15D0）

日志存储输出：

<22>Sep 17 19:12:14 postfix/smtp[18852]: 28D40A036B: to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9030A15D0)

{
  "@source": "syslog://192.244.100.42/",
  "@tags": [
    "_grokparsefailure"
  ],
  "@fields": {
    "priority": 13,
    "severity": 5,
    "facility": 1,
    "facility_label": "user-level",
    "severity_label": "Notice"
  },
  "@timestamp": "2013-09-17T17:12:06.958Z",
  "@source_host": "192.244.100.42",
  "@source_path": "/",
  "@message": "<22>Sep 17 19:12:14 postfix/smtp[18852]: 28D40A036B: to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9030A15D0)",
  "@type": "syslog"
}

filter {
    if [type] == "postfix" {
        grok {
            patterns_dir   => "/etc/logstash/patterns"
            match => { "message" => "%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} %{SYSLOGPROG}: %{POSTFIX_SMTP_DELIVERY}" }
        }
    }
}

{
    "postfix_queueid" => "28D40A036B",
    "@timestamp" => 2017-02-23T08:15:32.546Z,
    "postfix_smtp_response" => "250 2.0.0 Ok: queued as 9030A15D0",
    "port" => 50228,
    "postfix_keyvalue_data" => "to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent",
    "syslog5424_pri" => "22",
    "@version" => "1",
    "host" => "10.0.2.2",
    "pid" => "18852",
    "program" => "postfix/smtp",
    "message" => "<22>Sep 17 19:12:14 postfix/smtp[18852]: 28D40A036B: to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9030A15D0)"
}

{
“@source”：”syslog://192.244.100.42/",
“@tags”：[
“_grokparsefailure”
],
“@fields”：{
“优先”：13，
“严重性”：5，
“设施”：1，
“设施标签”：“用户级”，
“严重性标签”：“注意”
},
“@时间戳”：“2013-09-17T17:12:06.958Z”，
“@source_host”：“192.244.100.42”，
“@source_path”：“/”，
“@message:“9月17日19:12:14 postfix/smtp[18852]：28D40A036B:to=，relay=192.244.100.25[192.244.100.25]：25，delay=0.13，delays=0.01/0.01/0.09/0.02，dsn=2.0.0，status=sent（250 2.0.0.0确定：排队为9030A15D0）”，
@type:“系统日志”
}

我尝试使用grok解析器，但数据仍保留在

@message

字段中。我想将syslog解析器与正则表达式一起使用

---

解析

@message

字段需要遵循哪些步骤？

您的输出中有一个\u grokparsefailure表示解析日志时出现问题。您在配置中使用的grok过滤器是什么？

当我们现在使用Logstash 5.x时，grok的概念保持不变

不幸的是，Postfix在日志记录中有一些非常恼人的模式，就像少数人编写的一些模式一样，它们解释了您最终将在Postfix日志中看到的大部分数据。我将只使用其中的几个

关键是识别消息的组件，如果它们符合标准或在很大程度上流行，则可能已经为其编写了grok筛选器（例如syslog）。对于您不知道的消息组件，可以使用grok编写筛选器

让我们把信息分成几部分：

• Sep 17 19:12:14 postfix/smtp[18852]：

  ：这与RFC5424系统日志非常接近，但缺少ver（version）字段
  • SYSLOG5424PRI：优先级值
  • 系统日志时间戳：不言自明
  • SYSLOGPROG：应用程序的名称

• 28D40A036B:to=，relay=192.244.100.25[192.244.100.25]：25，delay=0.13，delays=0.01/0.01/0.09/0.02，dsn=2.0.0，status=sent（250 2.0.0.0确定：作为9030A15D0排队）

  ：这是要后缀的域特定信息
  • 后缀键值数据：用作另一个筛选器的组件，以匹配键值数据（例如中继=…，延迟=…）
  • 后缀\u队列ID：不言自明
  • POSTFIX\u KEYVALUE：组合POSTFIX\u QUEUEID和POSTFIX\u KEYVALUE\u数据
  • POSTFIX\u SMTP\u传递：使用POSTFIX\u KEYVALUE标识上述信息，直到status=，之后是SMTP响应

过滤器：

<22>Sep 17 19:12:14 postfix/smtp[18852]: 28D40A036B: to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9030A15D0)

{
  "@source": "syslog://192.244.100.42/",
  "@tags": [
    "_grokparsefailure"
  ],
  "@fields": {
    "priority": 13,
    "severity": 5,
    "facility": 1,
    "facility_label": "user-level",
    "severity_label": "Notice"
  },
  "@timestamp": "2013-09-17T17:12:06.958Z",
  "@source_host": "192.244.100.42",
  "@source_path": "/",
  "@message": "<22>Sep 17 19:12:14 postfix/smtp[18852]: 28D40A036B: to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9030A15D0)",
  "@type": "syslog"
}

filter {
    if [type] == "postfix" {
        grok {
            patterns_dir   => "/etc/logstash/patterns"
            match => { "message" => "%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} %{SYSLOGPROG}: %{POSTFIX_SMTP_DELIVERY}" }
        }
    }
}

{
    "postfix_queueid" => "28D40A036B",
    "@timestamp" => 2017-02-23T08:15:32.546Z,
    "postfix_smtp_response" => "250 2.0.0 Ok: queued as 9030A15D0",
    "port" => 50228,
    "postfix_keyvalue_data" => "to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent",
    "syslog5424_pri" => "22",
    "@version" => "1",
    "host" => "10.0.2.2",
    "pid" => "18852",
    "program" => "postfix/smtp",
    "message" => "<22>Sep 17 19:12:14 postfix/smtp[18852]: 28D40A036B: to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9030A15D0)"
}

将后缀模式保存在patterns_dir中

输出：

<22>Sep 17 19:12:14 postfix/smtp[18852]: 28D40A036B: to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9030A15D0)

{
  "@source": "syslog://192.244.100.42/",
  "@tags": [
    "_grokparsefailure"
  ],
  "@fields": {
    "priority": 13,
    "severity": 5,
    "facility": 1,
    "facility_label": "user-level",
    "severity_label": "Notice"
  },
  "@timestamp": "2013-09-17T17:12:06.958Z",
  "@source_host": "192.244.100.42",
  "@source_path": "/",
  "@message": "<22>Sep 17 19:12:14 postfix/smtp[18852]: 28D40A036B: to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9030A15D0)",
  "@type": "syslog"
}

filter {
    if [type] == "postfix" {
        grok {
            patterns_dir   => "/etc/logstash/patterns"
            match => { "message" => "%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} %{SYSLOGPROG}: %{POSTFIX_SMTP_DELIVERY}" }
        }
    }
}

{
    "postfix_queueid" => "28D40A036B",
    "@timestamp" => 2017-02-23T08:15:32.546Z,
    "postfix_smtp_response" => "250 2.0.0 Ok: queued as 9030A15D0",
    "port" => 50228,
    "postfix_keyvalue_data" => "to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent",
    "syslog5424_pri" => "22",
    "@version" => "1",
    "host" => "10.0.2.2",
    "pid" => "18852",
    "program" => "postfix/smtp",
    "message" => "<22>Sep 17 19:12:14 postfix/smtp[18852]: 28D40A036B: to=<test@gmail.com>, relay=192.244.100.25[192.244.100.25]:25, delay=0.13, delays=0.01/0.01/0.09/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9030A15D0)"
}

{
“postfix_queueid”=>“28D40A036B”，
“@timestamp”=>2017-02-23T08:15:32.546Z，
“postfix_smtp_response”=>“250 2.0.0正常：排队为9030A15D0”，
“端口”=>50228，
“postfix_keyvalue_data”=>“to=，relay=192.244.100.25[192.244.100.25]：25，延迟=0.13，延迟=0.01/0.01/0.09/0.02，dsn=2.0.0，状态=sent”，
“syslog5424_pri”=>“22”，
“@version”=>“1”，
“主机”=>“10.0.2.2”，
“pid”=>“18852”，
“程序”=>“后缀/smtp”，
“邮件”=>“9月17日19:12:14 postfix/smtp[18852]：28D40A036B:to=，中继=192.244.100.25[192.244.100.25]：25，延迟=0.13，延迟=0.01/0.01/0.09/0.02，dsn=2.0.0，状态=已发送（250 2.0.0.0确定：作为9030A15D0排队）”
}

上面所有的grok过滤器都是由其他人编写的，或者是由其他人编写的。幸运的是，许多人使用后缀，但很少有人为它编写过滤器，因为它相当复杂

一旦建立了日志库，您就可以使用日志库配置进行优化