elasticsearch logstash重试失败的操作,响应代码:429
锿:
节点1-节点10。10台机器 cpu:32核 jvm:60G 日志存储: logstash-node1,logstash-node2 cpu:32核 jvm 20G 每台机器上安装4个logstsh应用程序 数据源:18w/s-20w/s 数据格式:
elasticsearch logstash重试失败的操作,响应代码:429 锿:,
elasticsearch,logstash,
elasticsearch,Logstash,节点1-节点10。10台机器 cpu:32核 jvm:60G 日志存储: logstash-node1,logstash-node2 cpu:32核 jvm 20G 每台机器上安装4个logstsh应用程序 数据源:18w/s-20w/s 数据格式: "index" : { "number_of_shards" : 8, "number_of_replicas" : 0, "refresh_interval" : "30s" }
"index" : {
"number_of_shards" : 8,
"number_of_replicas" : 0,
"refresh_interval" : "30s"
}
"filed1" : {
"type": "integer"
},
"filed2" : {
"type": "integer"
},
"filed3" : {
"type": "long"
},
"filed4" : {
"type": "text"
},
"filed5":{
"type": "text"
},
"filed6":
{
"type": "text"
},
"fild7":{
"type": "date"
},
"field8":{
"type": "geo_point"
}
}
logstash.yml
//这默认为主机的CPU核心数
管道工人:20人
//每个输出插件实例应该使用多少工人
管道生产工人:20人
//在发送到筛选器+工作者之前,要从输入中检索多少事件
管道.批次.尺寸:9000
logstash kafka.conf
问:
现在速度为800w-900w/min
速度慢,
和logstash始终输出错误:
[2017-05-11T09:27:57,327][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$7@2393d3d9 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@63898a2e[Running, pool size = 32, active threads = 32, queued tasks = 50, completed tasks = 27163960]]"})
如何提高测井速度。厚度。es版本:5.2 logstash版本:5.2
[2017-05-11T09:27:57,327][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$7@2393d3d9 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@63898a2e[Running, pool size = 32, active threads = 32, queued tasks = 50, completed tasks = 27163960]]"})