elasticsearch Logstash不解析json
当我在Kibana中看到结果时,我看到JSON中没有字段,除此之外,elasticsearch Logstash不解析json,elasticsearch,logstash,kibana,elasticsearch,Logstash,Kibana,当我在Kibana中看到结果时,我看到JSON中没有字段,除此之外,消息字段只包含“状态”:“失败” 可以从json解析字段并在Kibana中显示它们吗? 我有以下配置: input { file { type => "json" path => "/home/logstash/test.json" codec => json sincedb_path => "/home/logstash/sincedb" } } output
消息
字段只包含“状态”:“失败”
可以从json解析字段并在Kibana中显示它们吗?
我有以下配置:
input {
file {
type => "json"
path => "/home/logstash/test.json"
codec => json
sincedb_path => "/home/logstash/sincedb"
}
}
output {
stdout {}
elasticsearch {
protocol => "http"
codec => "json"
host => "elasticsearch.dev"
port => "9200"
}
}
以及以下JSON文件:
[{"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182,"stop":1419621640491,"duration":17309},"severity":"NORMAL","status":"FAILED"},{"uid":"a88c89b377aca0c9","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182,"stop":1419621640634,"duration":17452},"severity":"NORMAL","status":"FAILED"},{"uid":"32c3f8b52386c85c","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623185,"stop":1419621640826,"duration":17641},"severity":"NORMAL","status":"FAILED"}]
对。您需要在配置中添加一个过滤器,类似这样
filter{
json{
source => "message"
}
}
文档中描述得很好
编辑
json编解码器似乎不喜欢传入数组。单个元素可用于此配置:
输入:
{"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182, "stop":1419621640491,"duration":17309 }, "severity":"NORMAL", "status":"FAILED" }
{
"message" => "{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }",
"@version" => "1",
"@timestamp" => "2015-02-26T23:25:12.011Z",
"host" => "emmet.local",
"uid" => "441d1d1dd296fe60",
"name" => "test_buylinks",
"title" => "Testbuylinks",
"time" => {
"start" => 1419621623182,
"stop" => 1419621640491,
"duration" => 17309
},
"severity" => "NORMAL",
"status" => "FAILED"
Trouble parsing json {:source=>"message", :raw=>"[{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }, {\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }]", :exception=>#<TypeError: can't convert Array into Hash>, :level=>:warn}
{
"message" => "[{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }, {\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }]",
"@version" => "1",
"@timestamp" => "2015-02-26T23:28:21.195Z",
"host" => "emmet.local",
"tags" => [
[0] "_jsonparsefailure"
]
}
日志存储结果:
{"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182, "stop":1419621640491,"duration":17309 }, "severity":"NORMAL", "status":"FAILED" }
{
"message" => "{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }",
"@version" => "1",
"@timestamp" => "2015-02-26T23:25:12.011Z",
"host" => "emmet.local",
"uid" => "441d1d1dd296fe60",
"name" => "test_buylinks",
"title" => "Testbuylinks",
"time" => {
"start" => 1419621623182,
"stop" => 1419621640491,
"duration" => 17309
},
"severity" => "NORMAL",
"status" => "FAILED"
Trouble parsing json {:source=>"message", :raw=>"[{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }, {\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }]", :exception=>#<TypeError: can't convert Array into Hash>, :level=>:warn}
{
"message" => "[{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }, {\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }]",
"@version" => "1",
"@timestamp" => "2015-02-26T23:28:21.195Z",
"host" => "emmet.local",
"tags" => [
[0] "_jsonparsefailure"
]
}
}
现在使用数组:
输入
[{"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182, "stop":1419621640491,"duration":17309 }, "severity":"NORMAL", "status":"FAILED" }, {"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182, "stop":1419621640491,"duration":17309 }, "severity":"NORMAL", "status":"FAILED" }]
结果:
{"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182, "stop":1419621640491,"duration":17309 }, "severity":"NORMAL", "status":"FAILED" }
{
"message" => "{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }",
"@version" => "1",
"@timestamp" => "2015-02-26T23:25:12.011Z",
"host" => "emmet.local",
"uid" => "441d1d1dd296fe60",
"name" => "test_buylinks",
"title" => "Testbuylinks",
"time" => {
"start" => 1419621623182,
"stop" => 1419621640491,
"duration" => 17309
},
"severity" => "NORMAL",
"status" => "FAILED"
Trouble parsing json {:source=>"message", :raw=>"[{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }, {\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }]", :exception=>#<TypeError: can't convert Array into Hash>, :level=>:warn}
{
"message" => "[{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }, {\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }]",
"@version" => "1",
"@timestamp" => "2015-02-26T23:28:21.195Z",
"host" => "emmet.local",
"tags" => [
[0] "_jsonparsefailure"
]
}
解析json时出现问题{:source=>“message”,:raw=>“[{\'uid\”:“441d1dd296fe60\”,“name\”:“test\u buylinks\”,“title\”:“Testbuylinks\”,“time\”:{“start\”:141962162162182,““stop\”:1419621640491,““duration\”:17309\,“severity\”,“severity\”:“NORMAL\,“status\”:“FAILED\”,“FAILED\”,“Testbuylinks\”,“,{'1d21d2d\”,“Testbuylinks\”,“feu\”,“test\”\“,”标题\“:”Testbuylinks \“,”时间\“:{”开始\“:1419621623182,\”停止\“:1419621640491,\”持续时间\“:17309},\”严重性\“:\”正常\“,\”状态\“:\”失败\“}]“,:exception=>#尝试使用json\u行
编解码器,而不是json
。这必须是最近添加的。在您的特定情况下,您首先需要将输出从json列表更改为换行分隔的json
此编解码器将解码以换行符分隔的流式JSON。对于
解码redis输入中的JSON负载例如,使用JSON
改为编解码器。编码将发出一个以
“\n”
读取包含JSON数组的文件要比实际困难得多
input {
exec {
command => "cat /path/file_containing_json_array.txt"
codec => "json"
interval => 3600
}
}
output {
stdout {
codec => rubydebug
}
}
我已经阅读了这些文档,但是过滤器没有帮助。结果是一样的。据我所知,这个定义说:消息包含可以用作哈希的有效JSON对象。这很好。但无论如何,我仍然没有在消息中看到有效的JSON。仅是部分。我想问题可能是,您的消息是一个数组,JSON编解码器在r处需要一个对象ootwow,也许我们可以将此报告为一个bug…无论如何,您是对的-filter是一个解决方案。似乎他们从源代码中意识到了这一点:#TODO(sissel):注意,这不会成功处理json列表#就像您的文本是“[1,2,3]”json.parse为您提供了一个数组(正确)不会合并成散列。如果有人需要,我们可以稍后修复。非常感谢您的帮助。现在一切都清楚了。无论如何,我已经在github上的票证中报告了