elasticsearch 来自Logstash的jsonparser异常

我正在尝试使用SQL中的logstash将数据插入elasticsearch。当使用配置文件执行此操作时，我会出现以下异常，logstash会不确定地运行。只有按ctrl+c，我才能停止logstash的执行

D:\Logstash\Logstash\logstash-5.1.1\bin>logstash -f D:\Logstash\Logstash\logstas
h-5.1.1\bin\Crash_data.conf
Using JAVA_HOME=C:\Program Files\Java\jre1.8.0_111 retrieved from C:\ProgramData
\Oracle\java\javapath\java.exe
Could not find log4j2 configuration at path /Logstash/Logstash/logstash-5.1.1/co
nfig/log4j2.properties. Using default config which logs to console
01:20:19.777 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - El
asticsearch pool URLs updated {:changes=>{:removed=>[], :added=>["http://10.64.1
03.61:5601"]}}
01:20:19.783 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Ru
nning health check to see if an Elasticsearch connection is working {:url=>#<URI
::HTTP:0x1dccc4ca URL:http://10.64.103.61:5601>, :healthcheck_path=>"/"}
01:20:20.141 [[main]-pipeline-manager] WARN  logstash.outputs.elasticsearch - Re
stored connection to ES instance {:url=>#<URI::HTTP:0x1dccc4ca URL:http://10.64.
103.61:5601>}
01:20:20.582 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Us
ing mapping template from {:path=>nil}
01:20:20.645 [[main]<jdbc] INFO  logstash.inputs.jdbc - (1.141000s) SELECT * fro
m [Device_crash_Reporting].[dbo].[Device_Crash_Data]
01:20:21.696 [[main]-pipeline-manager] ERROR logstash.outputs.elasticsearch - Fa
iled to install template. {:message=>"Unexpected character ('<' (code 60)): expe
cted a valid value (number, String, array, object, 'true', 'false' or 'null')\n
at [Source: [B@7c825cdd; line: 1, column: 2]", :class=>"LogStash::Json::ParserEr
ror"}
01:20:21.699 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Ne
w Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["10
.64.103.61:5601"]}
01:20:21.704 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeli
ne {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.b
atch.delay"=>5, "pipeline.max_inflight"=>250}
01:20:21.765 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main s
tarted
01:20:23.108 [Api Webserver] INFO  logstash.agent - Successfully started Logstas
h API endpoint {:port=>9600}
01:20:23.954 [[main]>worker1] ERROR logstash.outputs.elasticsearch - An unknown
error occurred sending a bulk request to Elasticsearch. We will retry indefinite
ly {:error_message=>"can't convert nil into Array", :error_class=>"TypeError", :
backtrace=>["org/jruby/RubyArray.java:1462:in `concat'", "D:/Logstash/Logstash/l
ogstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-j
ava/lib/logstash/outputs/elasticsearch/http_client.rb:94:in `join_bulk_responses
'", "org/jruby/RubyArray.java:1613:in `each'", "org/jruby/RubyEnumerable.java:85
2:in `inject'", "D:/Logstash/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gem
s/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/ht
tp_client.rb:92:in `join_bulk_responses'", "D:/Logstash/Logstash/logstash-5.1.1/
vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logsta
sh/outputs/elasticsearch/http_client.rb:88:in `bulk'", "D:/Logstash/Logstash/log
stash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-jav
a/lib/logstash/outputs/elasticsearch/common.rb:186:in `safe_bulk'", "D:/Logstash
/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsea
rch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:109:in `submit'", "D
:/Logstash/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-
elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:76:in `ret
rying_submit'", "D:/Logstash/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gem
s/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/co
mmon.rb:27:in `multi_receive'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-co
re/lib/logstash/output_delegator_strategies/shared.rb:12:in `multi_receive'", "D
:/Logstash/Logstash/logstash-5.1.1/logstash-core/lib/logstash/output_delegator.r
b:42:in `multi_receive'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-core/lib
/logstash/pipeline.rb:331:in `output_batch'", "org/jruby/RubyHash.java:1342:in `
each'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-core/lib/logstash/pipeline
.rb:330:in `output_batch'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-core/l
ib/logstash/pipeline.rb:288:in `worker_loop'", "D:/Logstash/Logstash/logstash-5.
1.1/logstash-core/lib/logstash/pipeline.rb:258:in `start_workers'"]}
01:20:24.338 [[main]>worker0] ERROR logstash.outputs.elasticsearch - An unknown
error occurred sending a bulk request to Elasticsearch. We will retry indefinite
ly {:error_message=>"can't convert nil into Array", :error_class=>"TypeError", :
backtrace=>["org/jruby/RubyArray.java:1462:in `concat'", "D:/Logstash/Logstash/l
ogstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-j
ava/lib/logstash/outputs/elasticsearch/http_client.rb:94:in `join_bulk_responses
'", "org/jruby/RubyArray.java:1613:in `each'", "org/jruby/RubyEnumerable.java:85
2:in `inject'", "D:/Logstash/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gem
s/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/ht
tp_client.rb:92:in `join_bulk_responses'", "D:/Logstash/Logstash/logstash-5.1.1/
vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logsta
sh/outputs/elasticsearch/http_client.rb:88:in `bulk'", "D:/Logstash/Logstash/log
stash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-jav
a/lib/logstash/outputs/elasticsearch/common.rb:186:in `safe_bulk'", "D:/Logstash
/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsea
rch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:109:in `submit'", "D
:/Logstash/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-
elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:76:in `ret
rying_submit'", "D:/Logstash/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gem
s/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/co
mmon.rb:27:in `multi_receive'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-co
re/lib/logstash/output_delegator_strategies/shared.rb:12:in `multi_receive'", "D
:/Logstash/Logstash/logstash-5.1.1/logstash-core/lib/logstash/output_delegator.r
b:42:in `multi_receive'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-core/lib
/logstash/pipeline.rb:331:in `output_batch'", "org/jruby/RubyHash.java:1342:in `
each'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-core/lib/logstash/pipeline
.rb:330:in `output_batch'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-core/l
ib/logstash/pipeline.rb:288:in `worker_loop'", "D:/Logstash/Logstash/logstash-5.
1.1/logstash-core/lib/logstash/pipeline.rb:258:in `start_workers'"]}
01:20:26.503 [[main]>worker1] ERROR logstash.outputs.elasticsearch - An unknown
error occurred sending a bulk request to Elasticsearch. We will retry indefinite
ly {:error_message=>"can't convert nil into Array", :error_class=>"TypeError", :
backtrace=>["org/jruby/RubyArray.java:1462:in `concat'", "D:/Logstash/Logstash/l
ogstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-j
ava/lib/logstash/outputs/elasticsearch/http_client.rb:94:in `join_bulk_responses
'", "org/jruby/RubyArray.java:1613:in `each'", "org/jruby/RubyEnumerable.java:85
2:in `inject'", "D:/Logstash/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gem
s/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/ht
tp_client.rb:92:in `join_bulk_responses'", "D:/Logstash/Logstash/logstash-5.1.1/
vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logsta
sh/outputs/elasticsearch/http_client.rb:88:in `bulk'", "D:/Logstash/Logstash/log
stash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-jav
a/lib/logstash/outputs/elasticsearch/common.rb:186:in `safe_bulk'", "D:/Logstash
/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsea
rch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:109:in `submit'", "D
:/Logstash/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-
elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:76:in `ret
rying_submit'", "D:/Logstash/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gem
s/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/co
mmon.rb:27:in `multi_receive'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-co
re/lib/logstash/output_delegator_strategies/shared.rb:12:in `multi_receive'", "D
:/Logstash/Logstash/logstash-5.1.1/logstash-core/lib/logstash/output_delegator.r
b:42:in `multi_receive'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-core/lib
/logstash/pipeline.rb:331:in `output_batch'", "org/jruby/RubyHash.java:1342:in `
each'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-core/lib/logstash/pipeline
.rb:330:in `output_batch'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-core/l
ib/logstash/pipeline.rb:288:in `worker_loop'", "D:/Logstash/Logstash/logstash-5.
1.1/logstash-core/lib/logstash/pipeline.rb:258:in `start_workers'"]}
01:20:26.847 [SIGINT handler] WARN  logstash.runner - SIGINT received. Shutting
down the agent.
^CTerminate batch job (Y/N)? 01:20:26.867 [LogStash::Runner] WARN  logstash.agen
t - stopping pipeline {:id=>"main"}
01:20:27.045 [[main]>worker0] ERROR logstash.outputs.elasticsearch - An unknown
error occurred sending a bulk request to Elasticsearch. We will retry indefinite
ly {:error_message=>"can't convert nil into Array", :error_class=>"TypeError", :
backtrace=>["org/jruby/RubyArray.java:1462:in `concat'", "D:/Logstash/Logstash/l
ogstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-j
ava/lib/logstash/outputs/elasticsearch/http_client.rb:94:in `join_bulk_responses
'", "org/jruby/RubyArray.java:1613:in `each'", "org/jruby/RubyEnumerable.java:85
2:in `inject'", "D:/Logstash/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gem
s/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/ht
tp_client.rb:92:in `join_bulk_responses'", "D:/Logstash/Logstash/logstash-5.1.1/
vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logsta
sh/outputs/elasticsearch/http_client.rb:88:in `bulk'", "D:/Logstash/Logstash/log
stash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-jav
a/lib/logstash/outputs/elasticsearch/common.rb:186:in `safe_bulk'", "D:/Logstash
/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsea
rch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:109:in `submit'", "D
:/Logstash/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-
elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:76:in `ret
rying_submit'", "D:/Logstash/Logstash/logstash-5.1.1/vendor/bundle/jruby/1.9/gem
s/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/co
mmon.rb:27:in `multi_receive'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-co
re/lib/logstash/output_delegator_strategies/shared.rb:12:in `multi_receive'", "D
:/Logstash/Logstash/logstash-5.1.1/logstash-core/lib/logstash/output_delegator.r
b:42:in `multi_receive'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-core/lib
/logstash/pipeline.rb:331:in `output_batch'", "org/jruby/RubyHash.java:1342:in `
each'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-core/lib/logstash/pipeline
.rb:330:in `output_batch'", "D:/Logstash/Logstash/logstash-5.1.1/logstash-core/l
ib/logstash/pipeline.rb:288:in `worker_loop'", "D:/Logstash/Logstash/logstash-5.
1.1/logstash-core/lib/logstash/pipeline.rb:258:in `start_workers'"]}
^C01:20:30.230 [SIGINT handler] FATAL logstash.runner - SIGINT received. Termina
ting immediately..


D:\Logstash\Logstash\logstash-5.1.1\bin>

---

您的

Crash_data.conf

配置文件也会有所帮助。Crash_data.conf如下所示。或者我已经用配置文件更新了我的问题。我认为在您的

elasticsearch

输出中，您需要

“10.64.103.61:9200”

而不是

“10.64.103.61:5601”

，后者是Kibana端点，而前者是Elasticsearch HTTP端点。您的数据库中是否有一个名为“message”的字段包含json？否则，json{}过滤器将不正确。

input {
jdbc {
    jdbc_driver_library => "C:\SQL JDBC Driver\sqljdbc_6.0\enu\sqljdbc42.jar"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://10.14.23.56:54287;instance:dev01;DatabaseName:Crash_Report;integratedSecurity=true"
    jdbc_user => "user"
    jdbc_password => "password"
    statement => "Select * from [Device_Crash_Reporting].[dbo].[Device_crash_Data]"
  }
}
filter {
json{
source=>"message"
}
}
output {
 elasticsearch {
    hosts => ["10.64.103.61:5601"] 
    index => "Crash_Data1"
 document_type => "Data"
  }

 stdout {codec => json_lines}
}