将旧日志从filebeat重新发送到logstash

提前感谢你的帮助。我想重新加载一些日志以自定义其他字段。我注意到filebeat配置中的注册表文件跟踪已拾取的文件。但是，如果我删除了该文件中的内容，我就不会恢复旧日志。我还尝试在注册表文件中更改源的时间戳，但没有成功。将旧日志从filebeat发送到logstash需要进行哪些更改

我怎样才能取回日志

更新：

这是tomcat容器中的最后一个日志：

2019-03-11 06:22:48 [Thread-4            ] DEBUG:   ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor Connection cache monitor in thread: Thread-4 shutting down for pool: WEB

这是filebeat获得的日志：

2019-03-14T16:18:50.377-0700    DEBUG   [publish]       pipeline/processor.go:308       Publish event: {
  "@timestamp": "2019-03-14T23:18:45.376Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.6.0"
  },
  "host": {
    "name": "tomcat",
    "architecture": "x86_64",
    "os": {
      "codename": "Core",
      "platform": "centos",
      "version": "7 (Core)",
      "family": "redhat",
      "name": "CentOS Linux"
    },
    "id": "6aaed308aa5a419f880c5e45eea65414",
    "containerized": true
  },
  "source": "/app/logs/WEB/WEB-rest-api/WEB-rest-api.log",
  "log": {
    "file": {
      "path": "/app/logs/WEB/WEB-rest-api/WEB-rest-api.log"
    }
  },
  "message": "2019-03-11 06:22:48 [Thread-4            ] DEBUG:   ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor Connection cache monitor in thread: Thread-4 shutting down for pool: WEB",
  "beat": {
    "name": "tomcat",
    "hostname": "tomcat",
    "version": "6.6.0"
  },
  "offset": 6771071,
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "meta": {
    "cloud": {
      "instance_name": "tomcat",
      "machine_type": "Standard_D8s_v3",
      "region": "CanadaCentral",
      "provider": "az",
      "instance_id": "6452bcf4-7f5d-4fc3-9f8e-5ea57f00724b"
    }
  }
}

这是Logstash接收的日志：

[2019-03-15T10:32:25,982][DEBUG][logstash.outputs.gelf    ] Sending GELF event {:event=>{"short_message"=>["2019-03-11 06:22:48 [Thread-4            ] DEBUG:   ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor Connection cache monitor in thread: Thread-4 shutting down for pool: WEB", " Connection cache monitor in thread: Thread-4 shutting down for pool: WEB"], "full_message"=>"2019-03-11 06:22:48 [Thread-4            ] DEBUG:   ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor Connection cache monitor in thread: Thread-4 shutting down for pool: WEB, Connection cache monitor in thread: Thread-4 shutting down for pool: WEB", "host"=>"{\"name\":\"tomcat\",\"os\":{\"name\":\"CentOS Linux\",\"version\":\"7 (Core)\",\"codename\":\"Core\"}}", "_source"=>"/app/logs/WEB/WEB-rest-api/WEB-rest-api.log", "_class"=>"ca.bc.gov.WEB.dbpool.WEBConnectionCacheMonitor, %{JAVACLASS}", "_tags"=>"beats_input_codec_plain_applied", "_beat_hostname"=>"tomcat", "_beat_name"=>"tomcat", "_meta_cloud"=>{}, "_log_file"=>{"path"=>"/app/logs/WEB/WEB-rest-api/WEB-rest-api.log"}, "level"=>6}}

Filebeat.yml：

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /apps/logs/WEB/web-api/web-api.log
    - /apps/logs/WEB/web-api/web-rest-api.log

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  # Ignore files which were modified more then the defined timespan in the past
  # Time strings like 2h (2 hours), 5m (5 minutes) can be used.
  ignore_older: 0

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  multiline.negate: true

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  multiline.match: after


#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
#setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

#============================= Elastic Cloud ==================================

# These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["log1.cgi-dev.ca:9200"]

  # Enabled ilm (beta) to use index lifecycle management instead daily indices.
  #ilm.enabled: false

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["log1.cgi-dev.ca:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash.crt"]

  # Certificate for SSL client authentication
  ##ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  ##ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================
# filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#xpack.monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
#xpack.monitoring.elasticsearch:

文件节拍配置示例#########################
#此文件是一个示例配置文件，仅突出显示最常见的
#选项。同一目录中的filebeat.reference.yml文件包含所有
#支持的选项包含更多注释。你可以把它作为参考。
#
#您可以在此处找到完整的配置参考：
# https://www.elastic.co/guide/en/beats/filebeat/index.html
#有关更多可用模块和选项，请参阅filebeat.reference.yml示例
#配置文件。
#==============================================文件节拍输入=============================
filebeat.inputs：
#每个-都是一个输入。大多数选项可以在输入级别设置，因此
#您可以对各种配置使用不同的输入。
#以下是特定于输入的配置。
-类型：原木
#更改为true以启用此输入配置。
已启用：true
#应该爬网和获取的路径。基于全局的路径。
路径：
-/apps/logs/WEB/WEB-api/WEB-api.log
-/apps/logs/WEB/WEB-api/WEB-rest-api.log
#排除行。要匹配的正则表达式列表。它删除了那些
#匹配列表中的任何正则表达式。
#排除_行：[“^DBG”]
#包括行。要匹配的正则表达式列表。它将导出所需的行
#匹配列表中的任何正则表达式。
#包括以下行：[“^ERR”、“^WARN”]
#排除文件。要匹配的正则表达式列表。Filebeat会删除
#正在匹配列表中的任何正则表达式。默认情况下，不会删除任何文件。
#排除_文件：['.gz$']
#可选的附加字段。这些田地可以自由采摘
#将附加信息添加到爬网日志文件以进行筛选
#字段：
#级别：调试
#审查：1
#忽略过去修改时间超过定义时间跨度的文件
#可以使用2小时（2小时）、5分钟（5分钟）等时间字符串。
忽略旧版本：0
###多行选项
#多行可用于跨多行的日志消息。这很常见
#用于Java堆栈跟踪或C行继续
#必须匹配的regexp模式。示例模式匹配以开头的所有行[
multiline.pattern:“^[0-9]{4}-[0-9]{2}-[0-9]{2}”
#定义pattern下的模式集是否应被否定。默认值为false。
多行。否定：真
#Match可以设置为“after”或“before”。它用于定义是否应将行附加到模式
#在否定之前或之后（不）匹配的，或者只要基于否定的模式不匹配。
#注：在Logstash中，After等同于previous，before等同于next
multiline.match:之后
#====================================================文件节拍模块===============================
filebeat.config.modules：
#配置加载的Glob模式
路径：${path.config}/modules.d/*.yml
#设置为true以启用配置重新加载
重新加载。已启用：false
#检查路径下文件的更改的时间段
#重新加载。周期：10秒
#===============================弹性搜索模板设置==========================
setup.template.settings：
index.number_的_碎片数：3
#index.codec：最佳压缩
#_source.enabled:false
#===================================================概述=====================================
#发布网络数据的发货人的名称。它可用于分组
#web界面中单个发货人发送的所有事务。
#姓名：
#托运人的标签包含在各自的字段中
#已发布交易记录。
#标签：[“服务-X”，“web层”]
#可选字段，您可以指定这些字段将附加信息添加到
#输出。
#字段：
#环境：登台
#=================================================仪表盘=====================================
#这些设置控制将示例仪表板加载到Kibana索引。加载
#默认情况下，仪表板处于禁用状态，可以通过设置
#选项，或者使用“-setup”CLI标志或“setup”命令。
#setup.dashboards.enabled:false
#从何处下载仪表板存档的URL。默认情况下，此URL
#具有基于节拍名称和版本计算的值。对于已发布
#版本，此URL指向artifacts.elastic.co上的仪表板存档
#网站。
#setup.dashboards.url：
#====================================================Kibana=====================================
#从Beats版本6.0.0开始，仪表板通过Kibana API加载。
#这需要Kibana端点配置。
#setup.kibana：
#Kibana主机
#方案和端口可以省略，并将设置为默认值（http和5601）
#如果指定了其他路径，则需要该方案：http://localhost:5601/path
#IPv6地址应始终定义为：https://[2001:db8:：1]：5601
#主机：“本地主机：5601”
#Kibana空间ID
#仪表板应加载到的Kibana空间的ID。默认情况下，
#将使用默认空间。
#space.id:
#=================================================弹性云==================================
#这些设置简化了将filebeat与弹性云结合使用的过程(https://cloud.elastic.co/).
#cloud.id设置覆盖“output.elasticsearch.hosts”和
#`setup.kibana.host`选项。
#你可以找到'cl'