Logstash:将URL参数放入哈希
我正在尝试使用Logstash和ElasticSearch来监视我的Apache Web服务器活动。现在,它运行得很好,但我需要更多关于我的请求字段的具体信息。 此时,我的日志存储配置为:Logstash:将URL参数放入哈希,url,filter,logstash,Url,Filter,Logstash,我正在尝试使用Logstash和ElasticSearch来监视我的Apache Web服务器活动。现在,它运行得很好,但我需要更多关于我的请求字段的具体信息。 此时,我的日志存储配置为: filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } grok { match => { "request" => [ "url", "%{URIPATH:url_path}%{URIPARAM:u
filter {
grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
grok { match => { "request" => [ "url", "%{URIPATH:url_path}%{URIPARAM:url_params}?" ]} }
urldecode{ field => "url_path" }
mutate { gsub => ["url_params","\?","" ] }
kv {
field_split => "&"
source => "url_params"
prefix => "url_param_"
}
date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] }
geoip { source => "clientip" }
useragent { source => "agent" }
}
{
"message" => "255.254.230.10 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/boreal%3A123456/status.php?pretty=true&test=boreal:%3A12345 HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"",
"@version" => "1",
"@timestamp" => "2013-12-11T08:01:45.000Z",
...
"request" => "/xampp/boreal%3A123456/status.php?pretty=true&test=boreal%3A12345",
"url_path" => "/xampp/boreal:123456/status.php",
"url_params" => "pretty=true&test=boreal%3A12345",
"url_param_pretty" => "true",
"url_param_test" => "boreal%3A12345",
...
}
获取基本apache日志:
255.254.230.10 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/boreal%3A123456/status.php?pretty=true&test=boreal%3A12345 HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"
第一次配置的结果是:
filter {
grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
grok { match => { "request" => [ "url", "%{URIPATH:url_path}%{URIPARAM:url_params}?" ]} }
urldecode{ field => "url_path" }
mutate { gsub => ["url_params","\?","" ] }
kv {
field_split => "&"
source => "url_params"
prefix => "url_param_"
}
date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] }
geoip { source => "clientip" }
useragent { source => "agent" }
}
{
"message" => "255.254.230.10 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/boreal%3A123456/status.php?pretty=true&test=boreal:%3A12345 HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"",
"@version" => "1",
"@timestamp" => "2013-12-11T08:01:45.000Z",
...
"request" => "/xampp/boreal%3A123456/status.php?pretty=true&test=boreal%3A12345",
"url_path" => "/xampp/boreal:123456/status.php",
"url_params" => "pretty=true&test=boreal%3A12345",
"url_param_pretty" => "true",
"url_param_test" => "boreal%3A12345",
...
}
并且(在梦境中),我希望url参数有以下响应:
{
...
"request" => "/xampp/boreal%3A123456/status.php?pretty=true&test=boreal%3A12345",
"url_path" => "/xampp/boreal:123456/status.php",
"url_params" => {
"pretty" => "true",
"url_param_test" => "boreal:12345"
},
...
}
我的口哨声
- url_参数将成为哈希数组
- 此散列的每个键都将是参数的名称
- 每个对应的值都将是urldecode值
- 我是否需要创建自己的插件(我还不熟悉ruby)李>
- 它是否存在一个现有的插件(我没有找到…可能是错误的搜索)
- 这是一种不用插件的方法吗
grok { match => { "request" => [ "url", "%{URIPATH:url_path}%{URIPARAM:url_params}?" ]} }
urldecode{ field => "url_path" }
mutate { gsub => ["url_params","\?","" ] }
kv {
field_split => "&"
source => "url_params"
target => "url_params_hash"
}
urldecode{ field => "url_params_hash" }
使用此解决方案,即使url参数字符串中有一个“&”(%26)字符,拆分也是正确的。使用
kv
过滤器,您的操作几乎是正确的。您需要稍微更改一下它的配置
您还需要在路径的另一个过滤器之后为url_参数
添加另一个urldecode
过滤器
urldecode{ field => "url_path" }
urldecode{ field => "url_params" }
mutate { gsub => ["url_params","\?","" ] }
kv {
field_split => "&"
source => "url_params"
target => "url_params_hash"
}
你会得到这样的结果:
{
"message" => "255.254.230.10 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/boreal%3A123456/status.php?pretty=true&test=boreal:%3A12345 HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"",
"@version" => "1",
"@timestamp" => "2013-12-11T08:01:45.000Z",
"url_params_hash" => {
"pretty" => "true",
"test" => "boreal:12345"
}
}
是 啊它工作得很好。非常感谢。我更改了顺序或urldecode:urldecode on“url_params”字段在“kv”过滤器之后完成。这样做,如果参数字符串中有一个“&”(%26)就没有问题了。太棒了,很高兴它有帮助!请注意,您应该编辑您的问题而不是我的答案:)是。。。坏点击对不起。您可以从答案中删除我的解决方案;-)