elasticsearch 未分析Grok筛选器
我正在尝试将Modsecurity-OSSEC日志添加到ELK。日志通过以下浏览器使用filebeat发送到ELK服务器:
elasticsearch 未分析Grok筛选器,
elasticsearch,logstash,elastic-stack,logstash-configuration,
elasticsearch,Logstash,Elastic Stack,Logstash Configuration,我正在尝试将Modsecurity-OSSEC日志添加到ELK。日志通过以下浏览器使用filebeat发送到ELK服务器: - paths: - /var/ossec/logs/alerts/alerts.log document_type: ossec-log multiline: pattern: '^\*\*' negate: true match: before max_lines: 20 timeout:
-
paths:
- /var/ossec/logs/alerts/alerts.log
document_type: ossec-log
multiline:
pattern: '^\*\*'
negate: true
match: before
max_lines: 20
timeout: 5s
在ELK服务器上,我在conf.d中创建了一个自定义日志存储配置,其中包含以下内容:
filter {
if [type] == "ossec-log" {
grok {
match => ["message", "(?m)\*\* Alert %{DATA:timestamp_seconds}:%{SPACE}%{WORD}?%{SPACE}\- %{DATA:ossec_group},\n%{YEAR} %{SYSLOGTIMESTAMP:syslog_timestamp}%{SPACE}%{HOSTNAME}\-\>%{DATA:reporting_source}\nRule: %{NONNEGINT:rule_number} \(level %{NONNEGINT:severity}\) \-\> '%{DATA:signature}'\nSrc IP: %{IP:src_ip}\n\[modsecurity\] \[client %{IP:src_ip}\] \[domain %{DATA:FQDN}\] \[%{NONNEGINT:DenyCode}\] \[%{DATA:LogLocation}\] \[file \"%{DATA:RuleConfig}\"\] \[line \"%{NONNEGINT:LineNumber}\"\] \[id \"%{NONNEGINT:RuleID}\"\] \[rev \"%{NONNEGINT:RevID}\"\] \[msg \"%{DATA:message}\"\] \[severity \"%{DATA:severity_text}\"\]%{GREEDYDATA}"]
}
}
}
它应该匹配的数据的结构如下:
** Alert 1471539036.4425320: mail - modsecurity,access_denied,
2016 Aug 18 18:50:36 myserver->/var/log/httpd/audit_log
Rule: 397989 (level 7) -> 'Atomicorp.com WAF Rules: MSIE 6.0 detected (Disable if you want to allow MSIE 6) '
Src IP: 127.127.127.127
[modsecurity] [client 127.127.127.127] [domain mydomain.org] [403] [/apache/20160818/20160818-1850/20160818-185034-YWhIuJT7g-sAAHaRtlsAAAAv] [file "/etc/httpd/modsecurity.d/20_asl_useragents.conf"] [line "369"] [id "397989"] [rev "1"] [msg "Atomicorp.com WAF Rules: MSIE 6.0 detected (Disable if you want to allow MSIE 6)"] [severity "WARNING"] Access denied with code 403 (phase 2). Match of "rx (MS Web Services Client Protocol|WormlyBot|webauth@cmcm\\.com)" against "REQUEST_HEADERS:User-Agent" required.
这在grokdebugger中有效,但不知何故,它在ELK中没有被解析。日志项显示在ELK中,类型正确(ossec日志),但它们似乎未被处理。日志条目已列出,但由于整行转储在“message”中,应用了一个名为“beats_input_codec_plain_”的标记,“grokparsefailure”,并且我在grok中定义的字段都不存在
我做错了什么
编辑190816:我通过将匹配括号切换为大括号并将逗号切换为=>(match=>{“message”=>“grok pattern”)。后来我还将filebeat config match指令从“before”切换为“after”