elasticsearch 无法删除grok筛选器不匹配的事件、日志存储、弹性搜索,elasticsearch,logstash,logstash-grok,elasticsearch,Logstash,Logstash Grok" /> elasticsearch 无法删除grok筛选器不匹配的事件、日志存储、弹性搜索,elasticsearch,logstash,logstash-grok,elasticsearch,Logstash,Logstash Grok" />
Fatal编程技术网
  • elasticsearch/
  • elasticsearch 无法删除grok筛选器不匹配的事件、日志存储、弹性搜索

elasticsearch 无法删除grok筛选器不匹配的事件、日志存储、弹性搜索

logstash

elasticsearch 无法删除grok筛选器不匹配的事件、日志存储、弹性搜索,elasticsearch,logstash,logstash-grok,elasticsearch,Logstash,Logstash Grok,我试图解析tomcat日志并将输出传递给elastic search。或多或少它工作得很好。当我看到弹性搜索索引数据时，它包含大量匹配的数据，标记字段为\u grokparsefailure。 这会导致大量重复的匹配数据。为了避免这种情况，我尝试在标记包含\u grokparsefailure时删除事件。这个配置写在grok过滤器下面的logstash.conf文件中。弹性搜索的输出仍然包含索引文档，其中包含带有\u grokparsefailure的标记。 如果grok失败，我不希望该匹配转到

我试图解析tomcat日志并将输出传递给elastic search。或多或少它工作得很好。当我看到弹性搜索索引数据时，它包含大量匹配的数据，标记字段为
\u grokparsefailure
。 这会导致大量重复的匹配数据。为了避免这种情况，我尝试在标记包含
\u grokparsefailure
时删除事件。这个配置写在grok过滤器下面的logstash.conf文件中。弹性搜索的输出仍然包含索引文档，其中包含带有
\u grokparsefailure
的标记。 如果grok失败，我不希望该匹配转到弹性搜索，因为它会导致弹性搜索中的重复数据

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 1.0,
    "hits" : [

 {

      "_index" : "logstash-2015.12.23",
      "_type" : "logs",
      "_id" : "J6CoEhKaSE68llz5nEbQSQ",
      "_score" : 1.0,
      "_source":{"message":"[2015-12-23 12:08:40,124] ERROR http-80-5_@{AF3AF784EC08D112D5D6FC92C78B5161,127.0.0.1,1450852688060} com.mmt.hotels.web.controllers.search.HotelsSearchController - Searching hotels for country IN, city DEL, checkin 28-03-2016, checkout 29-03-2016, roomstay 1e0e, No. of hotels returned is 6677 .","@version":"1","@timestamp":"2015-12-23T14:17:03.436Z","host":"ggn-37-97","path":"/opt/elasticSearch/logstash-1.4.2/input.log","tags":["_grokparsefailure"]}

    },

 {

      "_index" : "logstash-2015.12.23",
      "_type" : "logs",
      "_id" : "2XMc6nmnQJ-Bi8vxigyG8Q",
      "_score" : 1.0,
      "_source":{"@timestamp":"2015-12-23T14:17:02.894Z","message":"[2015-12-23 12:08:40,124] ERROR http-80-5_@{AF3AF784EC08D112D5D6FC92C78B5161,127.0.0.1,1450852688060} com.mmt.hotels.web.controllers.search.HotelsSearchController - Searching hotels for country IN, city DEL, checkin 28-03-2016, checkout 29-03-2016, roomstay 1e0e, No. of hotels returned is 6677 .","@version":"1","host":"ggn-37-97","path":"/opt/elasticSearch/logstash-1.4.2/input.log","country":"IN","city":"DEL","checkin":"28-03-2016","checkout":"29-03-2016","roomstay":"1e0e","hotelcount":"6677"}

},

 {

      "_index" : "logstash-2015.12.23",
      "_type" : "logs",
      "_id" : "fKLqw1LJR1q9YDG2yudRDw",
      "_score" : 1.0,
      "_source":{"@timestamp":"2015-12-23T14:16:12.684Z","message":"[2015-12-23 12:08:40,124] ERROR http-80-5_@{AF3AF784EC08D112D5D6FC92C78B5161,127.0.0.1,1450852688060} com.mmt.hotels.web.controllers.search.HotelsSearchController - Searching hotels for country IN, city DEL, checkin 28-03-2016, checkout 29-03-2016, roomstay 1e0e, No. of hotels returned is 6677 .","@version":"1","host":"ggn-37-97","path":"/opt/elasticSearch/logstash-1.4.2/input.log","country":"IN","city":"DEL","checkin":"28-03-2016","checkout":"29-03-2016","roomstay":"1e0e","hotelcount":"6677"}

    } ]
  }
}

logstash.conf
文件是：

input {

  file {

    path => "/opt/elasticSearch/logstash-1.4.2/input.log"
        codec => multiline {
                pattern => "^\["
                negate => true
                what => previous
        }
        start_position => "end"

  }

}

filter {

        grok {

    match => [
"message", "^\[%{GREEDYDATA}\] %{GREEDYDATA} Searching hotels for country %{GREEDYDATA:country}, city %{GREEDYDATA:city}, checkin %{GREEDYDATA:checkin}, checkout %{GREEDYDATA:checkout}, roomstay %{GREEDYDATA:roomstay}, No. of hotels returned is %{NUMBER:hotelcount} ."
    ]

  }

 if "_grokparsefailure"  in [tags]{     

        drop { }

    }

}

output {

file {
   path => "/opt/elasticSearch/logstash-1.4.2/output.log"
 }

 elasticsearch {
                cluster => "elasticsearchdev"
  }

}
弹性搜索响应
http://172.16.37.97:9200/logstash-2015.12.23/_search？pretty=true

下面给出的输出包含三个文档，其中第一个包含在_source->tags字段中的_grokparsefailure

我不希望它出现在这个输出中。所以可能需要限制它的日志存储，这样它就不会进入弹性搜索

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 1.0,
    "hits" : [

 {

      "_index" : "logstash-2015.12.23",
      "_type" : "logs",
      "_id" : "J6CoEhKaSE68llz5nEbQSQ",
      "_score" : 1.0,
      "_source":{"message":"[2015-12-23 12:08:40,124] ERROR http-80-5_@{AF3AF784EC08D112D5D6FC92C78B5161,127.0.0.1,1450852688060} com.mmt.hotels.web.controllers.search.HotelsSearchController - Searching hotels for country IN, city DEL, checkin 28-03-2016, checkout 29-03-2016, roomstay 1e0e, No. of hotels returned is 6677 .","@version":"1","@timestamp":"2015-12-23T14:17:03.436Z","host":"ggn-37-97","path":"/opt/elasticSearch/logstash-1.4.2/input.log","tags":["_grokparsefailure"]}

    },

 {

      "_index" : "logstash-2015.12.23",
      "_type" : "logs",
      "_id" : "2XMc6nmnQJ-Bi8vxigyG8Q",
      "_score" : 1.0,
      "_source":{"@timestamp":"2015-12-23T14:17:02.894Z","message":"[2015-12-23 12:08:40,124] ERROR http-80-5_@{AF3AF784EC08D112D5D6FC92C78B5161,127.0.0.1,1450852688060} com.mmt.hotels.web.controllers.search.HotelsSearchController - Searching hotels for country IN, city DEL, checkin 28-03-2016, checkout 29-03-2016, roomstay 1e0e, No. of hotels returned is 6677 .","@version":"1","host":"ggn-37-97","path":"/opt/elasticSearch/logstash-1.4.2/input.log","country":"IN","city":"DEL","checkin":"28-03-2016","checkout":"29-03-2016","roomstay":"1e0e","hotelcount":"6677"}

},

 {

      "_index" : "logstash-2015.12.23",
      "_type" : "logs",
      "_id" : "fKLqw1LJR1q9YDG2yudRDw",
      "_score" : 1.0,
      "_source":{"@timestamp":"2015-12-23T14:16:12.684Z","message":"[2015-12-23 12:08:40,124] ERROR http-80-5_@{AF3AF784EC08D112D5D6FC92C78B5161,127.0.0.1,1450852688060} com.mmt.hotels.web.controllers.search.HotelsSearchController - Searching hotels for country IN, city DEL, checkin 28-03-2016, checkout 29-03-2016, roomstay 1e0e, No. of hotels returned is 6677 .","@version":"1","host":"ggn-37-97","path":"/opt/elasticSearch/logstash-1.4.2/input.log","country":"IN","city":"DEL","checkin":"28-03-2016","checkout":"29-03-2016","roomstay":"1e0e","hotelcount":"6677"}

    } ]
  }
}
]您可以尝试在
输出部分测试
\u grokparsefailure
，如下所示：


output {
  if "_grokparsefailure" not in [tags] {
    file {
      path => "/opt/elasticSearch/logstash-1.4.2/output.log"
    }

    elasticsearch {
      cluster => "elasticsearchdev"
    }
  }
}



有时您可能有多个grok筛选器，其中一些筛选器可能会因某些事件而失败，但会因rest而通过，因此删除基于_grokparsefailure的事件不会解决问题

例如：


input
{
some input
}

filter
{
grok1 {extract ip to my_ip1}

grok2 {extract ip to my_ip2}

grok3 {extract ip to my_ip3}


}

output
{
  if "_grokparsefailure" not in [tags] { <-- This will not write to output if any single grok fails.
  some output
}
}



尝试过这个，但不起作用。Still output在标记中有_grokparsefailure。在尝试之前是否擦除了索引？i、 你确定你在看新文件而不是旧文件吗？是的，我看了。使用此命令删除索引。curl-XDELETE“好的，您能确保您没有查看过去索引中的文档吗，即
logstash-2015-12-23
，
logstash-2015-12-22
，等等文档仅适用于当天。我用一些dumy文本修改了我的input.log，以确保它能够接受最新的更改。此虚拟文本仍在输出中。