Logstash 从文件中获取JSON

日志存储5.2.1

我无法使用Logstash读取JSON文档。标准输出中没有文档

我像这样运行日志库：

./logstash-5.2.1/bin/logstash -f logstash-5.2.1/config/shakespeare.conf --config.reload.automatic 

日志存储配置：

input {
  file {
    path => "/home/trex/Development/Shipping_Data_To_ES/shakespeare.json"
    codec => json {}   
    start_position => "beginning"
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

此外，我还尝试了

字符集

：

...
codec => json {
  charset => "UTF-8"
}
...

此外，我还尝试在输入中使用/不使用

json

codec和过滤器：

...
filter {
  json {
    source => "message"
  }
}
...

启动后的日志存储控制台：

[2017-02-28T11:37:29,947][WARN ][logstash.agent           ] fetched new config for pipeline. upgrading.. {:pipeline=>"main", :config=>"input {\n  file {\n    path => \"/home/trex/Development/Shipping_Data_To_ES/shakespeare.json\"\n    codec => json {\n      charset => \"UTF-8\"\n    }\n    start_position => \"beginning\"\n  }\n}\n#filter {\n#  json {\n#    source => \"message\"\n#  }\n#}\noutput {\n  stdout {\n    codec => rubydebug\n  }\n}\n\n"}
[2017-02-28T11:37:29,951][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}
[2017-02-28T11:37:30,434][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-02-28T11:37:30,446][INFO ][logstash.pipeline        ] Pipeline main started
^C[2017-02-28T11:40:55,039][WARN ][logstash.runner          ] SIGINT received. Shutting down the agent.
[2017-02-28T11:40:55,049][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}
^C[2017-02-28T11:40:55,475][FATAL][logstash.runner          ] SIGINT received. Terminating immediately..
The signal INT is in use by the JVM and will not work correctly on this platform
[trex@Latitude-E5510 Shipping_Data_To_ES]$ ./logstash-5.2.1/bin/logstash -f logstash-5.2.1/config/shakespeare.conf --config.test_and_exit
^C[trex@Latitude-E5510 Shipping_Data_To_ES]$ ./logstash-5.2.1/bin/logstash -f logstash-5.2.1/config/shakespeare.conf --confireload.automatic
^C[trex@Latitude-E5510 Shipping_Data_To_ES]$ ./logstash-5.2.1/bin/logstash -f logstash-5.2.1/config/shakespeare.conf --config.reload.aumatic
Sending Logstash's logs to /home/trex/Development/Shipping_Data_To_ES/logstash-5.2.1/logs which is now configured via log4j2.properties
[2017-02-28T11:45:48,752][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-02-28T11:45:48,785][INFO ][logstash.pipeline        ] Pipeline main started
[2017-02-28T11:45:48,875][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

---

为什么Logstash不把我的JSON文档放在标准输出中

您是否尝试将该文件包含在您的

文件

输入中：

input {
  file {
    path => "/home/trex/Development/Shipping_Data_To_ES/shakespeare.json"
    type => "json"  <-- add this
    //codec => json {} <-- for the moment i'll comment this  
    start_position => "beginning"
  }
}

或如果您使用的是

编解码器

插件，请确保在

输入中包含这样的概要：

codec => "json"

或您可能也想尝试一下插件。希望这能派上用场。
您是否尝试将该文件包含在您的
文件
输入中：

input {
  file {
    path => "/home/trex/Development/Shipping_Data_To_ES/shakespeare.json"
    type => "json"  <-- add this
    //codec => json {} <-- for the moment i'll comment this  
    start_position => "beginning"
  }
}

或如果您使用的是
编解码器
插件，请确保在
输入中包含这样的概要：

codec => "json"

或您可能也想尝试一下插件。希望这能派上用场。
阅读JSON文件似乎很重要。我只有在添加了这个选项之后才能导入JSON。需要保持文件中的当前位置，以便在导入中断时能够从该位置恢复。我不需要任何位置跟踪，所以我只需将其设置为
/dev/null
，就可以了

基本工作日志存储配置：

input {
  file {
    path => ["/home/trex/Development/Shipping_Data_To_ES/shakespeare.json"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
output {
  stdout {
    codec => json_lines
  }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "shakespeare" 
  }  
}

读取JSON文件似乎很重要。我只有在添加了这个选项之后才能导入JSON。需要保持文件中的当前位置，以便在导入中断时能够从该位置恢复。我不需要任何位置跟踪，所以我只需将其设置为
/dev/null
，就可以了

基本工作日志存储配置：

input {
  file {
    path => ["/home/trex/Development/Shipping_Data_To_ES/shakespeare.json"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
output {
  stdout {
    codec => json_lines
  }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "shakespeare" 
  }  
}