Filter Logstash不'；不能用文件输入读取整行

我使用的是Logstash，我很难获得一个相当简单的配置来工作

input {
  file {
    path => "C:/path/test-data/*.log"
    start_position => beginning
    type => "usage_data"
  }
}

filter {
  if [type] == "usage_data" {
    grok {
      match => { "message" => "^\s*%{NUMBER:lineNumber}\s+%{TIMESTAMP_ISO8601:date},(?<value1>[A-Za-z0-9+/]+),(?<value2>[A-Za-z0-9+/]+),(?<value3>[A-Za-z0-9+/]+),(?<value4>[^,]+),(?<value5>[^\r]*)" }
    }
  }

  if "_grokparsefailure" not in [tags] {
    drop { }
  }
}

output {
  stdout { codec => rubydebug }
}

输出：

←[33mUsing milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.1/plugin-milestones {:level=>:w
arn}←[0m
{
       "message" => ",",
      "@version" => "1",
    "@timestamp" => "2014-11-20T09:16:08.591Z",
          "type" => "usage_data",
          "host" => "my-machine",
          "path" => "C:/path/test-data/monitor_20141116223000.log",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

如果我只解析

C:\path\test data\monitor\u 2014111623000.log

所有行都被读取，并且没有

grokparsefailure

。如果我删除

C:\path\test data\monitor\u 2014111623000.log

相同的

grokparsefailure

会在另一个日志文件中弹出：

{
       "message" => "atches in another context\r",
      "@version" => "1",
    "@timestamp" => "2014-11-20T09:14:04.779Z",
          "type" => "usage_data",
          "host" => "my-machine",
          "path" => "C:/path/test-data/monitor_20140829235900.log",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

特别是最后一个输出证明Logstash不会读取整行，或者尝试解释没有换行符的换行符。它总是在同一位置的同一条线上断开

也许我应该补充一下，日志文件包含

\n

作为行分隔符，我正在Windows上运行Logstash。然而，我并没有得到太多的错误，只是那一个。里面有很多线。当我删除

if“\u grokparsefailure”…

时，它们都会正确显示

我假设缓冲存在一些问题，但我不知道如何使其工作。有什么想法吗？

解决方法：

# diff -Nur /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb.orig /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb
--- /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb.orig       2015-02-25 10:46:06.916321816 +0700
+++ /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb    2015-02-12 18:39:34.943833909 +0700
@@ -86,7 +86,9 @@
           _read_file(path, &block)
           @files[path].close
           @files.delete(path)
-          @statcache.delete(path)
+          #@statcache.delete(path)
+          inode = @statcache.delete(path)
+          @sincedb[inode] = 0
         else
           @logger.warn("unknown event type #{event} for #{path}")
         end

---

我在Windows上运行时遇到了类似的问题。奇怪，谢谢，但你也把这件事告诉他们了吗？这将是很好的有一个正式版本。

# diff -Nur /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb.orig /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb
--- /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb.orig       2015-02-25 10:46:06.916321816 +0700
+++ /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb    2015-02-12 18:39:34.943833909 +0700
@@ -86,7 +86,9 @@
           _read_file(path, &block)
           @files[path].close
           @files.delete(path)
-          @statcache.delete(path)
+          #@statcache.delete(path)
+          inode = @statcache.delete(path)
+          @sincedb[inode] = 0
         else
           @logger.warn("unknown event type #{event} for #{path}")
         end