Logstash 在单个filebeat消息中的有效json行之间发送连续的无效json行

我有一个包含行分隔json对象和非json数据（stderr stacktraces）的文件

输出：

{
  "@timestamp": "2017-01-04T12:03:36.659Z",
  "beat": {
    "hostname": "...", "name": "...", "version": "5.1.1"
  },
  "input_type": "log",
  "json": {
    "event": "failed to download",
    "level": "info",
    "retry": 2,
    "timestamp": "20170104T17:10:41"
  },
  "offset": 285,
  "source": "/tmp/test.log",
  "type": "mytype"
}
{
  "@timestamp": "2017-01-04T12:03:36.659Z",
  "beat": {
    "hostname": "...", "name": "...", "version": "5.1.1"
  },
  "input_type": "log",
  "json": {
    "event": "Traceback (most recent call last):",
    "json_error": "Error decoding JSON: invalid character 'T' looking for beginning of value"
  },
  "offset": 320,
  "source": "/tmp/test.log",
  "type": "mytype"
}

我希望将所有非json行合并到一个新的json行中 信息

使用multiline，我尝试了以下方法

filebeat.prospectors:
  - input_type: log
    document_type: mytype
    json:
      message_key: event
      add_error_key: true
    paths:
        - /tmp/*.log
    multiline:
      pattern: '^{'
      negate: true
      match: after

output:
  console:
    pretty: true

  file:
    path: "/tmp/filebeat"
    filename: filebeat

但它似乎不起作用。它对

event

key的值执行多行规则，该值在

json.message\u key

中指定

从这个角度，我理解了为什么会发生这种情况

json.message\u键

-

用于应用行筛选和多行设置的JSON键。 此键必须为顶级，且其值必须为字符串，否则为 被忽略了。如果未定义文本键，则行过滤和 无法使用多行要素

是否有其他方法将连续的非json行合并到单个消息中

---

我希望在将整个堆栈跟踪发送到logstash之前捕获它。

Filebeat在JSON解析后应用多行分组，因此多行模式不能基于构成JSON对象的字符（例如

{

）

在Filebeat中，还有另一种进行JSON解析的方法，即JSON解析发生在多行分组之后，因此您的模式可以包含JSON对象字符。您需要Filebeat 5.2（即将发布）因为

target

字段已添加到处理器中，因此您可以指定解码的json字段将添加到事件中的位置

filebeat.prospectors:
- paths: [input.txt]
  multiline:
    pattern: '^({|Traceback)'
    negate:  true
    match:   after

processors:
- decode_json_fields:
    when.regexp:
      message: '^{'
    fields: message
    target:
- drop_fields:
    when.regexp:
      message: '^{'
    fields: message

我使用Golang游乐场测试了多行模式

Filebeat生成以下输出（使用上面提供的日志行作为输入）。（我使用了主分支的构建。）

---

Filebeat在JSON解析后应用多行分组，因此多行模式不能基于构成JSON对象的字符（例如，

{

）

在Filebeat中，还有另一种进行JSON解析的方法，即JSON解析发生在多行分组之后，因此您的模式可以包含JSON对象字符。您需要Filebeat 5.2（即将发布）因为

target

字段已添加到处理器中，因此您可以指定解码的json字段将添加到事件中的位置

filebeat.prospectors:
- paths: [input.txt]
  multiline:
    pattern: '^({|Traceback)'
    negate:  true
    match:   after

processors:
- decode_json_fields:
    when.regexp:
      message: '^{'
    fields: message
    target:
- drop_fields:
    when.regexp:
      message: '^{'
    fields: message

我使用Golang游乐场测试了多行模式

Filebeat生成以下输出（使用上面提供的日志行作为输入）。（我使用了主分支的构建。）

filebeat.prospectors:
- paths: [input.txt]
  multiline:
    pattern: '^({|Traceback)'
    negate:  true
    match:   after

processors:
- decode_json_fields:
    when.regexp:
      message: '^{'
    fields: message
    target:
- drop_fields:
    when.regexp:
      message: '^{'
    fields: message

{"@timestamp":"2017-01-05T20:34:18.862Z","beat":{"hostname":"host.example.com","name":"host.example.com","version":"5.2.0-SNAPSHOT"},"event":"failed to download","input_type":"log","level":"info","offset":95,"retry":0,"source":"input.txt","timestamp":"20170104T17:10:39","type":"log"}
{"@timestamp":"2017-01-05T20:34:18.862Z","beat":{"hostname":"host.example.com","name":"host.example.com","version":"5.2.0-SNAPSHOT"},"event":"failed to download","input_type":"log","level":"info","offset":190,"retry":1,"source":"input.txt","timestamp":"20170104T17:10:40","type":"log"}
{"@timestamp":"2017-01-05T20:34:18.862Z","beat":{"hostname":"host.example.com","name":"host.example.com","version":"5.2.0-SNAPSHOT"},"event":"failed to download","input_type":"log","level":"info","offset":285,"retry":2,"source":"input.txt","timestamp":"20170104T17:10:41","type":"log"}
{"@timestamp":"2017-01-05T20:34:18.862Z","beat":{"hostname":"host.example.com","name":"host.example.com","version":"5.2.0-SNAPSHOT"},"input_type":"log","message":"Traceback (most recent call last):\n  File \"a.py\", line 12, in \u003cmodule\u003e\n    foo()\n  File \"a.py\", line 10, in foo\n    bar()\n  File \"a.py\", line 4, in bar\n    raise Exception(\"This was unexpected\")\nException: This was unexpected","offset":511,"source":"input.txt","type":"log"}
{"@timestamp":"2017-01-05T20:34:18.862Z","beat":{"hostname":"host.example.com","name":"host.example.com","version":"5.2.0-SNAPSHOT"},"event":"failed to download","input_type":"log","level":"info","offset":606,"retry":3,"source":"input.txt","timestamp":"20170104T17:10:42","type":"log"}
{"@timestamp":"2017-01-05T20:34:18.862Z","beat":{"hostname":"host.example.com","name":"host.example.com","version":"5.2.0-SNAPSHOT"},"event":"failed to download","input_type":"log","level":"info","offset":702,"retry":4,"source":"input.txt","timestamp":"20170104T17:10:43","type":"log"}