Logstash(解析Nginx日志时的grokparsefailure)
我正在尝试使用Logstash解析nginx日志,除了用包含nginx$remote\u用户的行获取这个Logstash(解析Nginx日志时的grokparsefailure),nginx,
elasticsearch,logstash,logstash-grok,logstash-configuration,Nginx,
elasticsearch,Logstash,Logstash Grok,Logstash Configuration,我正在尝试使用Logstash解析nginx日志,除了用包含nginx$remote\u用户的行获取这个\u grokparsefailure标记外,一切看起来都很好。当$remote_user为“-”(未指定$remote_user时的默认值)时,Logstash将执行此任务,但使用真正的$remote_用户,如user@gmail.com它失败并放置一个\u grokparsefailure标签: 127.0.0.1--[17/Feb/2017:23:14:08+0100]“GET/favi
\u grokparsefailure
标记外,一切看起来都很好。当$remote_user为“-”(未指定$remote_user时的默认值)时,Logstash将执行此任务,但使用真正的$remote_用户,如user@gmail.com
它失败并放置一个\u grokparsefailure
标签:
127.0.0.1--[17/Feb/2017:23:14:08+0100]“GET/favicon.ico HTTP/1.1“302 169”“”Mozilla/5.0(X11;Linux
x86_64)AppleWebKit/537.36(KHTML,类似Gecko)Chrome/56.0.2924.87
Safari/537.36“
====>工作正常
127.0.0.1-jemlifathi@gmail.com[17/Feb/2017:23:14:07+0100]“GET/trainer/templates/home.tmpl.html HTTP/1.1”304 0
“”“Mozilla/5.0(X11;Linux x86_64)
AppleWebKit/537.36(KHTML,比如Gecko)Chrome/56.0.2924.87
Safari/537.36“
====>\u grokparsefailure
标记并无法分析日志行
我正在使用此配置文件:
input {
file {
path => "/home/dev/node/training-hub/logs/access_log"
start_position => "beginning"
sincedb_path => "/dev/null"
ignore_older => 0
type => "logs"
}
}
filter {
if[type] == "logs" {
mutate {
gsub => ["message", "::ffff:", ""]
}
grok {
match=> [
"message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}",
"message" , "%{COMMONAPACHELOG}+%{GREEDYDATA:extra_fields}"
]
overwrite=> [ "message" ]
}
mutate {
convert=> ["response", "integer"]
convert=> ["bytes", "integer"]
convert=> ["responsetime", "float"]
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
date {
match=> [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
remove_field=> [ "timestamp" ]
}
useragent {
source=> "agent"
}
}
}
output { elasticsearch { hosts => "localhost:9200" } }
在使用许多值测试输出后,我意识到Logstash无法解析包含此类
$remote\u user
的日志行,因为它不是有效的用户名(电子邮件地址),因此我添加了mutate gsub
过滤器以删除@和邮件地址的其余部分,从而获得有效的$remote\u user
gsub=>[“消息”,
(((:::((::::::(::::::::25[0-5)2[0-5[0-4[0-4[0-4[0-4][0-9[0-4[0-4[0-4[0-4[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-5[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[x0b\x0c\x0e-\x7f])+)]
[“,”[“]
现在,它运行良好在使用许多值测试输出后,我意识到Logstash无法解析包含此类
$remote\u user
的日志行,因为它不是有效的用户名(电子邮件地址)因此,我添加了一个mutate gsub
过滤器,以删除@和邮件地址的其余部分,从而获得一个有效的$remote\u用户
gsub=>[“消息”,
(((:::((::::::(::::::::25[0-5)2[0-5[0-4[0-4[0-4[0-4][0-9[0-4[0-4[0-4[0-4[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-5[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[0-9[x0b\x0c\x0e-\x7f])+)]
[“,”[“]
现在,它工作得很好