elasticsearch 删除注册表时重复的日志事件

我目前正在进行PoC ELK安装，我想重新发送在Filebeat中注册的文件的每一行日志，以便进行测试

我就是这么做的：

我停下来

我通过Kibana删除Logstash中的索引

我删除Filebeat注册表文件

我开始打拍子

在Kibana中，我可以看到有两倍于日志行的事件，并且我还可以看到每个事件都复制了一次

为什么呢

Filebeat日志：

2017-05-05T14:25:16+02:00 INFO Setup Beat: filebeat; Version: 5.2.2
2017-05-05T14:25:16+02:00 INFO Max Retries set to: 3
2017-05-05T14:25:16+02:00 INFO Activated logstash as output plugin.
2017-05-05T14:25:16+02:00 INFO Publisher name: anonymized
2017-05-05T14:25:16+02:00 INFO Flush Interval set to: 1s
2017-05-05T14:25:16+02:00 INFO Max Bulk Size set to: 2048
2017-05-05T14:25:16+02:00 INFO filebeat start running.
2017-05-05T14:25:16+02:00 INFO No registry file found under: /var/lib/filebeat/registry. Creating a new registry file.
2017-05-05T14:25:16+02:00 INFO Loading registrar data from /var/lib/filebeat/registry
2017-05-05T14:25:16+02:00 INFO States Loaded from registrar: 0
2017-05-05T14:25:16+02:00 INFO Loading Prospectors: 1
2017-05-05T14:25:16+02:00 INFO Prospector with previous states loaded: 0
2017-05-05T14:25:16+02:00 INFO Loading Prospectors completed. Number of prospectors: 1
2017-05-05T14:25:16+02:00 INFO All prospectors are initialised and running with 0 states to persist
2017-05-05T14:25:16+02:00 INFO Starting Registrar
2017-05-05T14:25:16+02:00 INFO Start sending events to output
2017-05-05T14:25:16+02:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-05-05T14:25:16+02:00 INFO Starting prospector of type: log
2017-05-05T14:25:16+02:00 INFO Harvester started for file: /some/where/anonymized.log
2017-05-05T14:25:46+02:00 INFO Non-zero metrics in the last 30s: registrar.writes=2 libbeat.logstash.publish.read_bytes=54 libbeat.logstash.publish.write_bytes=32390 libbeat.logstash.published_and_acked_events=578 filebeat.harvester.running=1 registar.states.current=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.publisher.published_events=578 publish.events=579 filebeat.harvester.started=1 registrar.states.update=579 filebeat.harvester.open_files=1
2017-05-05T14:26:16+02:00 INFO No non-zero metrics in the last 30s

---

删除注册表文件造成了问题

Filebeat管理浏览器（内存中）和注册表文件（保存在磁盘中）的文件状态和事件确认

请阅读文档

您可以自己管理每个事件的_id字段，以便任何重复的事件（无论出于何种原因，甚至在生产环境中）在elasticsearch中都不会有两个，而是会更新事件

在日志存储管道配置文件中创建以下配置

#if your logs don't have a unique ID, use the following to generate one
fingerprint{
        #with the message field or choose other(s) that can give you a  uniqueID
        source => ["message"]
        target => "LogID"
        key    => "something"
        method => "MD5"
        concatenate_sources => true
           }
 #in your output section 
    elasticsearch{
                     hosts => ["localhost:9200"]
                     document_id => "%{LogID}"
                     index => "yourindex"
        }