Logstash-存储RabbitMQ日志-多行

我已经用麋鹿大约六个月了，到目前为止效果很好。我使用的是logstash版本6.2.3。 RabbitMQ构成了我的分布式系统的核心（RabbitMQ本身是分布式的），因此跟踪RabbitMQ的日志非常重要。 这个论坛上的大多数其他对话似乎都使用RabbitMQ作为输入/输出阶段，但我只想监视日志。 我发现的唯一问题是RabbitMQ具有多行日志记录，如下所示：

=WARNING REPORT==== 19-Nov-2017::06:53:14 ===
closing AMQP connection <0.27161.0> (...:32799 -> ...:5672, vhost: '/', user: 'worker'):
client unexpectedly closed TCP connection

=WARNING REPORT==== 19-Nov-2017::06:53:18 ===
closing AMQP connection <0.22410.0> (...:36656 -> ...:5672, vhost: '/', user: 'worker'):
client unexpectedly closed TCP connection

=WARNING REPORT==== 19-Nov-2017::06:53:19 ===
closing AMQP connection <0.26045.0> (...:55427 -> ...:5672, vhost: '/', user: 'worker'):
client unexpectedly closed TCP connection

=WARNING REPORT==== 19-Nov-2017::06:53:20 ===
closing AMQP connection <0.5484.0> (...:47740 -> ...:5672, vhost: '/', user: 'worker'):
client unexpectedly closed TCP connection

但是，当我将其保存到conf文件并重新启动logstash时，会出现以下错误：

[2018-04-04T07:01:57,308][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-04-04T07:01:57,316][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-04-04T07:01:57,841][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.3"}
[2018-04-04T07:01:57,973][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-04-04T07:01:58,037][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 3, column 15 (byte 54) after filter {\n    if [type] == \"rabbitmq\" {\n        codec ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

你知道这是什么问题吗

---

谢谢，

您不能将编解码器用作过滤器插件。编解码器只能在带有编解码器配置选项的输入或输出插件中使用（请参阅）

---

您必须将多行编解码器放在生成rabbitmq日志的输入插件中。

如果您使用filebeat将日志从rabbitmq服务器发送到logstash，您应该在那里配置。答案确实是肯定的。目标是将以日期以外的内容开头的行与以日期开头的前一行合并。这就是为什么：

multiline.pattern: '^\d{4}-\d{2}-\d{2}'
multiline.negate: true
multiline.match: after

注意：我以前尝试合并任何以空格字符开头的行，但没有成功，因为并非所有警告或错误消息都以空格开头

完成文件节拍输入（7.5.2格式）

日志存储模式：

# RabbitMQ
RABBITMQDATE %{MONTHDAY}-%{MONTH}-%{YEAR}::%{HOUR}:%{MINUTE}:%{SECOND}
RABBITMQLINE (?m)=%{DATA:severity} %{DATA}==== %{RABBITMQDATE:timestamp} ===\n%{GREEDYDATA:message}

我确信他们有很好的理由以这种奇怪的方式登录RMQ

3.7.x

，但不了解他们，这确实让我们的生活很艰难

filebeat:
  inputs:
  - exclude_lines:
    - 'Failed to publish events caused by: EOF'
    fields:
      type: rabbitmq
    fields_under_root: true
    paths:
    - /var/log/rabbitmq/*.log
    tail_files: false
    timeout: 60s
    type: log
    multiline.pattern: '^\d{4}-\d{2}-\d{2}'
    multiline.negate: true
    multiline.match: after

# RabbitMQ
RABBITMQDATE %{MONTHDAY}-%{MONTH}-%{YEAR}::%{HOUR}:%{MINUTE}:%{SECOND}
RABBITMQLINE (?m)=%{DATA:severity} %{DATA}==== %{RABBITMQDATE:timestamp} ===\n%{GREEDYDATA:message}