Logstash:如果输出失败,将事件发送到其他位置
提供以下日志存储管道:Logstash:如果输出失败,将事件发送到其他位置,logstash,logstash-configuration,Logstash,Logstash Configuration,提供以下日志存储管道: input { generator { lines => [ '{"name" : "search", "product" : { "module" : "search" , "name" : "api"}, "data" : { "query" : "toto"}}', '{"name" : "user_interaction", "product" : { "module" : "search" ,
input
{
generator
{
lines => [
'{"name" : "search", "product" : { "module" : "search" , "name" : "api"}, "data" : { "query" : "toto"}}',
'{"name" : "user_interaction", "product" : { "module" : "search" , "name" : "front"}, "data" : { "query" : "toto"}}',
'{"name" : "search", "product" : { "module" : "search" , "name" : "api"}, "data" : { "query" : "toto"}}',
'{"hello": "world"}',
'{"name" :"wrong data", "data" : "I am wrong !"}',
'{"name" :"wrong data", "data" : { "hello" : "world" }}'
]
codec => json
count => 1
}
}
filter
{
mutate
{
remove_field => ["sequence", "host", "@version"]
}
}
output
{
elasticsearch
{
hosts => ["elasticsearch:9200"]
index => "events-dev6-test"
document_type => "_doc"
manage_template => false
}
stdout
{
codec => rubydebug
}
}
elasticsearch对此索引有严格的映射,因此,一些事件会给出400错误“映射设置为严格,不允许在[data]中动态引入[hello]”
(这是正常的)
如何将失败的事件发送到其他地方(文本日志或其他elasticsearch索引)(这样我就不会丢失事件)?介绍了Logstash 6.2,可用于执行所需操作。您需要在logstash.yml
中启用死信队列。enable:true
然后将其作为输入进行处理:
input {
dead_letter_queue {
path => "/path/to/data/dead_letter_queue"
commit_offsets => true
pipeline_id => "main"
}
}
output {
file {
path => ...
codec => line { format => "%{message}"}
}
}
在6.2之前,我不相信有一种方法可以实现你想要的