Filter Logstash Grok筛选器Apache访问日志

我四处寻找，但找不到有效的解决方案。我尝试在Logstash配置文件中使用Grok过滤器来过滤Apache访问日志文件。日志消息如下所示：

{“message”：“00.00.0.000---[dd/mm/YYYY:hh:mm:ii+0000]”GET/index.html HTTP/1.1 \“200 00”}。

此时，我只能使用

grok{match=>[“message”，“%{ip:client_ip}”]}

过滤客户端ip

我要筛选：

- The GET method, 
- requested page (index.html), 
- HTTP/1.1\, 
- server response 200
- the last number 00 after 200 inside the message body

请注意，这些都不适用于我：

grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } 

或

---

使用Grok调试器获得与日志格式完全匹配的结果。这是唯一的办法

使用以下方法：

filter {
    grok {
            match => { "message" => "%{COMMONAPACHELOG}" }
    }
}

从您的模式中可以看出，CombinedPacheLog将失败，因为缺少一些组件：

COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

---

您可以使用

CombinedAppachelog

模式

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}

例如，考虑此示例Apache日志

111.222.333.123主页-[01/Feb/1998:01:08:46-0800]“GET/bannerad/ad.htm HTTP/1.0”200 28083 “”“Mozilla/4.01 （麦金塔；I；PPC）

上面的过滤器会产生

{
  "clientip": [
    [
      "111.222.333.123"
    ]
  ],
  "HOSTNAME": [
    [
      "111.222.333.123"
    ]
  ],
  "IP": [
    [
      null
    ]
  ],
  "IPV6": [
    [
      null
    ]
  ],
  "IPV4": [
    [
      null
    ]
  ],
  "ident": [
    [
      "HOME"
    ]
  ],
  "USERNAME": [
    [
      "HOME",
      "-"
    ]
  ],
  "auth": [
    [
      "-"
    ]
  ],
  "timestamp": [
    [
      "01/Feb/1998:01:08:46 -0800"
    ]
  ],
  "MONTHDAY": [
    [
      "01"
    ]
  ],
  "MONTH": [
    [
      "Feb"
    ]
  ],
  "YEAR": [
    [
      "1998"
    ]
  ],
  "TIME": [
    [
      "01:08:46"
    ]
  ],
  "HOUR": [
    [
      "01"
    ]
  ],
  "MINUTE": [
    [
      "08"
    ]
  ],
  "SECOND": [
    [
      "46"
    ]
  ],
  "INT": [
    [
      "-0800"
    ]
  ],
  "verb": [
    [
      "GET"
    ]
  ],
  "request": [
    [
      "/bannerad/ad.htm"
    ]
  ],
  "httpversion": [
    [
      "1.0"
    ]
  ],
  "BASE10NUM": [
    [
      "1.0",
      "200",
      "28083"
    ]
  ],
  "rawrequest": [
    [
      null
    ]
  ],
  "response": [
    [
      "200"
    ]
  ],
  "bytes": [
    [
      "28083"
    ]
  ],
  "referrer": [
    [
      ""http://www.referrer.com/bannerad/ba_intro.htm""
    ]
  ],
  "QUOTEDSTRING": [
    [
      ""http://www.referrer.com/bannerad/ba_intro.htm"",
      ""Mozilla/4.01 (Macintosh; I; PPC)""
    ]
  ],
  "agent": [
    [
      ""Mozilla/4.01 (Macintosh; I; PPC)""
    ]
  ]
}

可以在这里测试

---

您能提供原始日志吗？您的原始日志不是{“message”：“00.00.0.000---[dd/mm/YYYY:hh:mm:ii+0000]”GET/index.html HTTP/1.1 \“200 00”}@Bem Lim，我现在找到了解决方案。谢谢<代码>#日志格式SYSLOGBASE%{SYSLOGTIMESTAMP:timestamp}（？：{SYSLOGFACILITY}）%{SYSLOGHOST:logsource}%{SYSLOGPROG}:commonapacachelog%{IPORHOST:clientip}%{USER:ident}%{USER:auth}\[%{HTTPDATE timestamp}\]（？：%{WORD verb}%{verb}{NOTSPACE request}{request}（？HTTP/%{NUMBER:httpversion}）}数据：请求数：{12400%}CombinedAppachelog%{COMMONAPACHELOG}%{QS:referer}%{QS:agent}Source-

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}

{
  "clientip": [
    [
      "111.222.333.123"
    ]
  ],
  "HOSTNAME": [
    [
      "111.222.333.123"
    ]
  ],
  "IP": [
    [
      null
    ]
  ],
  "IPV6": [
    [
      null
    ]
  ],
  "IPV4": [
    [
      null
    ]
  ],
  "ident": [
    [
      "HOME"
    ]
  ],
  "USERNAME": [
    [
      "HOME",
      "-"
    ]
  ],
  "auth": [
    [
      "-"
    ]
  ],
  "timestamp": [
    [
      "01/Feb/1998:01:08:46 -0800"
    ]
  ],
  "MONTHDAY": [
    [
      "01"
    ]
  ],
  "MONTH": [
    [
      "Feb"
    ]
  ],
  "YEAR": [
    [
      "1998"
    ]
  ],
  "TIME": [
    [
      "01:08:46"
    ]
  ],
  "HOUR": [
    [
      "01"
    ]
  ],
  "MINUTE": [
    [
      "08"
    ]
  ],
  "SECOND": [
    [
      "46"
    ]
  ],
  "INT": [
    [
      "-0800"
    ]
  ],
  "verb": [
    [
      "GET"
    ]
  ],
  "request": [
    [
      "/bannerad/ad.htm"
    ]
  ],
  "httpversion": [
    [
      "1.0"
    ]
  ],
  "BASE10NUM": [
    [
      "1.0",
      "200",
      "28083"
    ]
  ],
  "rawrequest": [
    [
      null
    ]
  ],
  "response": [
    [
      "200"
    ]
  ],
  "bytes": [
    [
      "28083"
    ]
  ],
  "referrer": [
    [
      ""http://www.referrer.com/bannerad/ba_intro.htm""
    ]
  ],
  "QUOTEDSTRING": [
    [
      ""http://www.referrer.com/bannerad/ba_intro.htm"",
      ""Mozilla/4.01 (Macintosh; I; PPC)""
    ]
  ],
  "agent": [
    [
      ""Mozilla/4.01 (Macintosh; I; PPC)""
    ]
  ]
}