elasticsearch 日志存储中的CSV过滤器_csvparsefailure“；错误

我问了艾勒另一个问题，我认为可能与这个问题有关： 我认为它相关的原因是因为在前面的问题中，kibana没有显示JSON解析器的结果，JSON解析器的“PROGRAM”字段为“mfd_status”。现在我正在改变我做事的方式，删除了JSON解析器，以防它可能会干扰一些东西，但我仍然没有显示任何带有“mfd_状态”的日志

csv 
{
    columns => ["unixTime", "unixTime2", "FACILITY_NUM", "LEVEL_NUM", "PROGRAM", "PID", "MSG_FULL"]
    source => "message"
    separator => "  "
}

在上一个问题的过滤器中，我使用了两个grok过滤器，现在我将它们替换为csv过滤器。我还有两个日期和一个指纹过滤器，但我认为它们与这个问题无关

日志消息示例：

“1452564798.76\t1452496397.00\t1\t4\tkernel\t\t[6252.000246]声纳：声纳写入（）：正在等待…”

输出：

        "unixTime" => "1452564798.76",
       "unixTime2" => "1452496397.00",
    "FACILITY_NUM" => "1",
       "LEVEL_NUM" => "4",
         "PROGRAM" => "kernel",
             "PID" => nil,
        "MSG_FULL" => "[ 6252.000246] sonar: sonar_write(): waiting...",
       "TIMESTAMP" => "2016-01-12T02:13:18.760Z",
"TIMESTAMP_second" => "2016-01-11T07:13:17.000Z"

       "tags" => [
    [0] "_csvparsefailure"

“1452564804.57\t1452496403.00\t1\t7\tmfd\U状态\t\T00800F08Cf00\textra\t{“日期”：1452543203，“主机”：ABCD1234，“inet\：[“169.254.42.207/16”，“10.8.207.176/32”，“172.22.42.207/16”，“fb0:[“U:1280x800p-60”，“32]””

输出：

        "unixTime" => "1452564798.76",
       "unixTime2" => "1452496397.00",
    "FACILITY_NUM" => "1",
       "LEVEL_NUM" => "4",
         "PROGRAM" => "kernel",
             "PID" => nil,
        "MSG_FULL" => "[ 6252.000246] sonar: sonar_write(): waiting...",
       "TIMESTAMP" => "2016-01-12T02:13:18.760Z",
"TIMESTAMP_second" => "2016-01-11T07:13:17.000Z"

       "tags" => [
    [0] "_csvparsefailure"

在日志中显示kernel/mfd_状态之后，不应该有更多的除味器，它应该全部放在MSG_FULL字段下

---

总之，为什么我的一条日志消息解析正确，而另一条没有？此外，即使它没有正确解析，它仍然应该将它发送到elasticsearch，只使用空字段，我想，为什么它也不这样做呢？

你几乎做得很好，你需要在CSV过滤器中覆盖另外两个参数，这两行将被正确解析

第一个是

skip\u empty\u columns=>true

，因为在第二个日志行中有一个空字段，需要忽略它

第二个是

quote\u char=>“”

（或除双引号

“

”以外的任何内容），因为JSON包含双引号

csv {
    columns => ["unixTime", "unixTime2", "FACILITY_NUM", "LEVEL_NUM", "PROGRAM", "PID", "MSG_FULL"]
    source => "message"
    separator => "  "
    skip_empty_columns => true
    quote_char => "'"
}

使用此选项，您的第一个日志行解析为：

{
         "message" => "1452564798.76\\t1452496397.00\\t1\\t4\\tkernel\\t\\t[ 6252.000246] sonar: sonar_write(): waiting...",
        "@version" => "1",
      "@timestamp" => "2016-01-12T04:21:34.051Z",
            "host" => "iMac.local",
        "unixTime" => "1452564798.76",
       "unixTime2" => "1452496397.00",
    "FACILITY_NUM" => "1",
       "LEVEL_NUM" => "4",
         "PROGRAM" => "kernel",
        "MSG_FULL" => "[ 6252.000246] sonar: sonar_write(): waiting..."
}

{
         "message" => "1452564804.57\\t1452496403.00\\t1\\t7\\tmfd_status\\t\\t00800F08CFB0\\textra\\t{\\\"date\\\":1452543203,\\\"host\\\":\\\"ABCD1234\\\",\\\"inet\\\":[\\\"169.254.42.207/16\\\",\\\"10.8.207.176/32\\\",\\\"172.22.42.207/16\\\"],\\\"fb0\\\":[\\\"U:1280x800p-60\\\",32]}",
        "@version" => "1",
      "@timestamp" => "2016-01-12T04:21:07.974Z",
            "host" => "iMac.local",
        "unixTime" => "1452564804.57",
       "unixTime2" => "1452496403.00",
    "FACILITY_NUM" => "1",
       "LEVEL_NUM" => "7",
         "PROGRAM" => "mfd_status",
        "MSG_FULL" => "00800F08CFB0",
         "column8" => "extra",
         "column9" => "{\\\"date\\\":1452543203,\\\"host\\\":\\\"ABCD1234\\\",\\\"inet\\\":[\\\"169.254.42.207/16\\\",\\\"10.8.207.176/32\\\",\\\"172.22.42.207/16\\\"],\\\"fb0\\\":[\\\"U:1280x800p-60\\\",32]}"
}

第二条日志行解析为：

{
         "message" => "1452564798.76\\t1452496397.00\\t1\\t4\\tkernel\\t\\t[ 6252.000246] sonar: sonar_write(): waiting...",
        "@version" => "1",
      "@timestamp" => "2016-01-12T04:21:34.051Z",
            "host" => "iMac.local",
        "unixTime" => "1452564798.76",
       "unixTime2" => "1452496397.00",
    "FACILITY_NUM" => "1",
       "LEVEL_NUM" => "4",
         "PROGRAM" => "kernel",
        "MSG_FULL" => "[ 6252.000246] sonar: sonar_write(): waiting..."
}

{
         "message" => "1452564804.57\\t1452496403.00\\t1\\t7\\tmfd_status\\t\\t00800F08CFB0\\textra\\t{\\\"date\\\":1452543203,\\\"host\\\":\\\"ABCD1234\\\",\\\"inet\\\":[\\\"169.254.42.207/16\\\",\\\"10.8.207.176/32\\\",\\\"172.22.42.207/16\\\"],\\\"fb0\\\":[\\\"U:1280x800p-60\\\",32]}",
        "@version" => "1",
      "@timestamp" => "2016-01-12T04:21:07.974Z",
            "host" => "iMac.local",
        "unixTime" => "1452564804.57",
       "unixTime2" => "1452496403.00",
    "FACILITY_NUM" => "1",
       "LEVEL_NUM" => "7",
         "PROGRAM" => "mfd_status",
        "MSG_FULL" => "00800F08CFB0",
         "column8" => "extra",
         "column9" => "{\\\"date\\\":1452543203,\\\"host\\\":\\\"ABCD1234\\\",\\\"inet\\\":[\\\"169.254.42.207/16\\\",\\\"10.8.207.176/32\\\",\\\"172.22.42.207/16\\\"],\\\"fb0\\\":[\\\"U:1280x800p-60\\\",32]}"
}

---

非常好。谢谢