elasticsearch 使用grok过滤Logstash中的Apache错误日志
我需要使用grok过滤apache错误日志。 请帮助我创建我无法创建的模式 我的示例日志:
elasticsearch 使用grok过滤Logstash中的Apache错误日志,
elasticsearch,logstash,logstash-grok,logstash-configuration,
elasticsearch,Logstash,Logstash Grok,Logstash Configuration,我需要使用grok过滤apache错误日志。 请帮助我创建我无法创建的模式 我的示例日志: 2020-10-07T01:21:26.403-0400 ERROR [reload] cfgfile/list.go:96 Error creating runner from config: Error getting config for fileset system/auth: Error interpreting the template of the inp$ 202
2020-10-07T01:21:26.403-0400 ERROR [reload] cfgfile/list.go:96 Error creating runner from config: Error getting config for fileset system/auth: Error interpreting the template of the inp$
2020-10-07T01:21:36.404-0400 ERROR [reload] cfgfile/list.go:96 Error creating runner from config: Error getting config for fileset system/auth: Error interpreting the template of the inp$
2020-10-07T01:21:38.925-0400 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://IP:5044)): dial tcp IP:5044: i/o timeout
2020-10-07T01:21:38.925-0400 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://IP:5044)) with 26743 reconnect attempt(s)
2020-10-07T01:21:38.925-0400 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2020-10-07T01:21:38.925-0400 INFO [publish] pipeline/retry.go:191 done
2020-10-07T01:21:38.925-0400 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2020-10-07T01:21:38.925-0400 INFO [publish] pipeline/retry.go:168 done
我通过grok模式了解,我们可以使用以下这些,但我不了解如何在grok模式中使用这些:
# Error logs
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{GREEDYDATA:message}
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}
谁能帮帮我吗!提前谢谢 处理样本数据后,此grok模式必须有效:
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601}%{SPACE}%{LOGLEVEL}(%{SPACE}\[%{WORD:action}\])?%{SPACE}%{WORD:package}/%{WORD:class}.go:%{INT:line:number}%{SPACE}%{GREEDYDATA:message}$" }
}
}
您的数据不完全是http,因此需要一个自定义模式,我想我的grok在没有空间的情况下一定更易于阅读,我建议您使用mutate gsub来统一空间(请在我的解决方案中使用名为“message”的最后一句话)
您有关于此模式和其他模式的更多详细信息。请在您的问题中添加日志示例。您使用哪个版本的logstash?@karanshah添加了示例日志。@YLR我正在使用logstash 7.9.2So筛选错误日志。我可以使用:match=>{“message”=>“%{HTTPD20_ERRORLOG}”?您可以在此处尝试模式:HTTPD20_ERRORLOG与问题中共享的日志示例不匹配。@ylr建议的过滤器模式应该可以工作。