Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/apache-kafka/3.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Logstash Kafka输入插件的低性能,除了度量之外没有op输出_Logstash_Apache Kafka - Fatal编程技术网

Logstash Kafka输入插件的低性能,除了度量之外没有op输出

Logstash Kafka输入插件的低性能,除了度量之外没有op输出,logstash,apache-kafka,Logstash,Apache Kafka,测试环境如下: CPU: Intel L5640 2.26 GHz 6 cores * 2 EA Memory: SAMSUNG PC3-10600R 4 GB * 4 EA HDD: TOSHIBA SAS 10,000 RPM 300 GB * 6 EA OS: CentOS release 6.6 (Final) Logstash 2.3.4 我使用了以下配置: input { kafka { zk_connect => '1.2.3.4:2181' top

测试环境如下:

CPU: Intel L5640 2.26 GHz 6 cores * 2 EA
Memory: SAMSUNG PC3-10600R 4 GB * 4 EA
HDD: TOSHIBA SAS 10,000 RPM 300 GB * 6 EA

OS: CentOS release 6.6 (Final)

Logstash 2.3.4
我使用了以下配置:

input {
  kafka {
    zk_connect => '1.2.3.4:2181'
    topic_id => 'some-log'
    consumer_threads => 1
  }
}

filter {
  metrics {
    meter => "events"
    add_tag => "metric"
  }
}

output {
  if "metric" in [tags] {
    stdout {
      codec => line {
        format => "Count: %{[events][count]}"
      }
    }
  }
}
我得到了以下结果:

./bin/logstash -f some-log-kafka.conf                                
Settings: Default pipeline workers: 24
Pipeline main started
Count: 9614
Count: 23080
Count: 37087
Count: 50815
Count: 64517
Count: 78296
Count: 91977
Count: 105990
./bin/logstash -f impression-log-kafka.conf 
Settings: Default pipeline workers: 24
Pipeline main started
Count: 9599
Count: 23254
Count: 37253
Count: 51029
Count: 64881
Count: 78868
Count: 92663
Count: 106267
默认的刷新间隔是5秒,所以看起来大约是每5秒14K(每秒2.8K)

consumer\u threads
设置为10时,我得到了以下结果:

./bin/logstash -f some-log-kafka.conf                                
Settings: Default pipeline workers: 24
Pipeline main started
Count: 9614
Count: 23080
Count: 37087
Count: 50815
Count: 64517
Count: 78296
Count: 91977
Count: 105990
./bin/logstash -f impression-log-kafka.conf 
Settings: Default pipeline workers: 24
Pipeline main started
Count: 9599
Count: 23254
Count: 37253
Count: 51029
Count: 64881
Count: 78868
Count: 92663
Count: 106267
它看起来越来越多的
消费者线程
没有多大区别

根据我简单的无操作消费者基准测试,我预计大约为3万,至少为10万,但这只是预期性能的1/10

如何提高其性能

补充意见: 对于Kafka客户端Java库,我使用引导服务器,而对于Logstash Kafka输入插件,我使用ZooKeeper(引导服务器没有选项)。我不确定这是否会导致巨大的差异