Logstash-从卡夫卡到ES的数据

使用logstash 5.0.0，将kafka源作为输入->在Elasticsearch中获取数据并生成输出。（ElasticSearch 5.0.0版）

日志存储配置：

input{
kafka{
bootstrap_servers => "XXX.XXX.XX.XXX:9092","XXX.XXX.XX.XXX:9092","XXX.XXX.XX.XXX:9092"
topics => ["a-data","f-data","n-data"]
group_id => "sound"
auto_offset_reset => "earliest"
consumer_threads => 2
}
}
filter{
json{
source => "message"
}
}
output {
elasticsearch {
hosts => [ "XXX.XXX.XX.XXX:9200" ]
}
}

当我运行下面的配置时，我得到以下错误

$ ./logstash -f sound.conf
Sending Logstash logs to /logstash-5.0.0/logs which is now configured vi   a log4j2.properties.
[2017-01-17T10:53:29,273][ERROR][logstash.agent           ] fetched an invalid c   onfig {:config=>"input{\nkafka{\nbootstrap_servers => \"XX.XXX.XXX.XX:9092\",\"XXX.XXX.XX.XXX:9092\",\"XXX.XXX.XX.XXX:9092\"\ntopics => [\"a-data\",\"f-data\   ",\"n-data\"]\ngroup_id => \"sound\"\nauto_offset_reset => \"earliest\"\nc   onsumer_threads => 2\n}\n}\nfilter{\njson{\nsource => \"message\"\n}\n}\noutput    {\nelasticsearch {\nhosts => [ \"XX.XX.XXX.XX:9200\" ]\n}\n}\n\n", :reason=>"Ex   pected one of #, {, } at line 3, column 40 (byte 54) after input{\nkafka{\nboots   trap_servers => \"XX.XX.XXX.XX:9092\""}

---

有人能帮我做这个配置吗。

你的

主题不应该是一个数组，在那里你把值作为一个散列插入：

topics => ["a-data","f-data","n-data"] <-- try changing this line

topics=>[“a-data”、“f-data”、“n-data”]在您的
kafka
输入中，我没有看到5.0中包含最后三个概要。达斯·维德-我不明白。最后三个概要？我指的是这三个：组\u id=>“声音”自动\u偏移量\u重置=>“最早”消费线程=>2我排除了这三行，也尝试了。你能帮我做这个吗。我尝试删除组\u id=>“声音”自动\u偏移量\u重置=>“最早”消费者\u线程=>2。。但似乎不起作用。有人能帮忙吗？