elasticsearch 日志存储和弹性搜索:在一个值中拆分值
刚刚开始使用logstash和弹性搜索 下面是我的日志: 2015-09-09 16:02:23 GET/NeedA/some1/some2/some3/NeedB/some4/NeedC f=json-127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_5)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/44.0.2403.157+Safari/537.36 200 373 554 46 使用下面的配置文件,我可以单独获得url: /需要A/some1/some2/some3/NeedB/some4/NeedCelasticsearch 日志存储和弹性搜索:在一个值中拆分值,elasticsearch,logstash,grok,elasticsearch,Logstash,Grok,刚刚开始使用logstash和弹性搜索 下面是我的日志: 2015-09-09 16:02:23 GET/NeedA/some1/some2/some3/NeedB/some4/NeedC f=json-127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_5)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/44.0.2403.157+Safari/537.36 200 373 554 46 使
filter {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:method} %{URIPATH:url} %{NOTSPACE:querystring} %{NOTSPACE:username} %{IPORHOST:ipaddress} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:scstatus} %{NUMBER:scbytes:int} %{NUMBER:csbytes:int} %{NUMBER:timetaken:int}"]
}
date {
match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
timezone => "Etc/UCT"
}
}
问题:
如何将NeedA、NeedB和NeedC从/NeedA/some1/some2/some3/NeedB/some4/NeedC中分离出来,并将其作为弹性搜索中的不同字段
grok {
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:method} \/%{WORD:fieldA}\/.*\/.*\/.*\/%{WORD:fieldB}\/.*\/%{WORD:fieldC} %{NOTSPACE:querystring} %{NOTSPACE:username} %{IPORHOST:ipaddress} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:scstatus} %{NUMBER:scbytes:int} %{NUMBER:csbytes:int} %{NUMBER:timetaken:int}"]
}
在您的程序中,只需将%{URIPATH:url}
替换为\/%{WORD:fieldA}\/.\/.\/.\/.\/.\/%{WORD:fieldB}\/.\/.\/%{WORD:fieldC}
输出结果:
{
"message" => "2015-09-09 16:02:23 GET /NeedA/some1/some2/some3/NeedB/some4/NeedC f=json - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_5)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/44.0.2403.157+Safari/537.36 http://localhost:3000/ 200 373 554 46",
"@version" => "1",
"@timestamp" => "2015-09-09T16:02:23.000Z",
"host" => "MyHost.local",
"path" => "/path/of/test.log",
"log_timestamp" => "2015-09-09 16:02:23",
"method" => "GET",
"fieldA" => "NeedA",
"fieldB" => "NeedB",
"fieldC" => "NeedC",
"querystring" => "f=json",
"username" => "-",
"ipaddress" => "127.0.0.1",
"useragent" => "Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_5)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/44.0.2403.157+Safari/537.36",
"referer" => "http://localhost:3000/",
"scstatus" => "200",
"scbytes" => 373,
"csbytes" => 554,
"timetaken" => 46
}
问候,,
阿兰