Cassandra 卡桑德拉的读操作大幅增加
我正在外面的测试环境中和卡桑德拉一起玩。我在集群中有3个C*（版本3.11）节点和3个列族，具有复制因子2和SizeTieredCompactionStrategy。每个节点都部署到AWS r4.large实例，其中有两个EBS gp2磁盘—100GB用于提交日志，300GB用于数据。 2列族是只写的，我们的应用程序在这些表中写了很多内容（大约每秒800-1000个插入），并且从不读取它们。第三列族是一个计数器表，它的使用方式类似于“增量和获取”模式：大约每秒200个增量和获取。在整个测试过程中，每个计数器的更新频率不高，大约为30次 5天内一切正常，但其中一个节点上的读取操作数/秒开始增长到>1000 iops，达到磁盘iops限制： 当达到限制时，从计数器表中选择的响应时间从2位毫秒跳到秒，并且读取io在第二个节点上开始增长 以下是nodetool cfstats的输出：Cassandra 卡桑德拉的读操作大幅增加,cassandra,cassandra-3.0,Cassandra,Cassandra 3.0,我正在外面的测试环境中和卡桑德拉一起玩。我在集群中有3个C*（版本3.11）节点和3个列族，具有复制因子2和SizeTieredCompactionStrategy。每个节点都部署到AWS r4.large实例，其中有两个EBS gp2磁盘—100GB用于提交日志，300GB用于数据。 2列族是只写的，我们的应用程序在这些表中写了很多内容（大约每秒800-1000个插入），并且从不读取它们。第三列族是一个计数器表，它的使用方式类似于“增量和获取”模式：大约每秒200个增量和获取。在整个测试过程中
Keyspace : geo
Read Count: 51744711
Read Latency: 12.840126352082631 ms.
Write Count: 201887146
Write Latency: 0.02199966441647553 ms.
Pending Flushes: 0
Table: gpm
SSTable count: 5
Space used (live): 5127907030
Space used (total): 5127907030
Space used by snapshots (total): 0
Off heap memory used (total): 102252166
SSTable Compression Ratio: 0.41318162863841235
Number of keys (estimate): 32583794
Memtable cell count: 51001
Memtable data size: 3072328
Memtable off heap memory used: 0
Memtable switch count: 176
Local read count: 51530318
Local read latency: NaN ms
Local write count: 97595727
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 80436992
Bloom filter off heap memory used: 80436952
Index summary off heap memory used: 21046326
Compression metadata off heap memory used: 768888
Compacted partition minimum bytes: 87
Compacted partition maximum bytes: 149
Compacted partition mean bytes: 112
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
任何帮助都将不胜感激。是否有任何分析工具可以帮助我了解问题所在？您可能已经到了必须压缩sstables的地步。您的调试日志应该有压缩数据，或者您可能会在磁盘上发现一个sstable，它的文件时间与磁盘读取速率的下降大致一致（压缩完成）。您可能已经到了必须压缩sstable的程度。您的调试日志应该有压缩数据，或者您可能会在磁盘上发现一个sstable，它的文件时间大致与磁盘读取速率的下降（压缩完成）一致。
Execute CQL3 query | 2017-08-28 16:06:44.110000 | 172.31.16.220 | 0 | 172.31.16.220
Parsing select * from geo.gpm where usr = '57469' and lat = 55.617 and lon = 37.509; [Native-Transport-Requests-1] | 2017-08-28 16:06:44.110000 | 172.31.16.220 | 187 | 172.31.16.220
Preparing statement [Native-Transport-Requests-1] | 2017-08-28 16:06:44.110000 | 172.31.16.220 | 411 | 172.31.16.220
reading data from /172.31.32.220 [Native-Transport-Requests-1] | 2017-08-28 16:06:44.110000 | 172.31.16.220 | 736 | 172.31.16.220
Sending READ message to cassandra-test-skywalker.aws.local/172.31.32.220 [MessagingService-Outgoing-cassandra-test-skywalker.aws.local/172.31.32.220-Small] | 2017-08-28 16:06:44.111000 | 172.31.16.220 | 1184 | 172.31.16.220
READ message received from /172.31.16.220 [MessagingService-Incoming-/172.31.16.220] | 2017-08-28 16:06:44.111000 | 172.31.32.220 | 35 | 172.31.16.220
Executing single-partition query on gpm [ReadStage-3] | 2017-08-28 16:06:44.117000 | 172.31.32.220 | 5675 | 172.31.16.220
Acquiring sstable references [ReadStage-3] | 2017-08-28 16:06:44.117000 | 172.31.32.220 | 5745 | 172.31.16.220
Skipped 0/5 non-slice-intersecting sstables, included 0 due to tombstones [ReadStage-3] | 2017-08-28 16:06:44.117000 | 172.31.32.220 | 5812 | 172.31.16.220
Key cache hit for sstable 85 [ReadStage-3] | 2017-08-28 16:06:44.117000 | 172.31.32.220 | 5906 | 172.31.16.220
Key cache hit for sstable 170 [ReadStage-3] | 2017-08-28 16:06:44.117000 | 172.31.32.220 | 5963 | 172.31.16.220
Key cache hit for sstable 191 [ReadStage-3] | 2017-08-28 16:06:44.117000 | 172.31.32.220 | 5996 | 172.31.16.220
Key cache hit for sstable 212 [ReadStage-3] | 2017-08-28 16:06:44.117001 | 172.31.32.220 | 6028 | 172.31.16.220
Bloom filter allows skipping sstable 217 [ReadStage-3] | 2017-08-28 16:06:44.118000 | 172.31.32.220 | 6069 | 172.31.16.220
Merged data from memtables and 4 sstables [ReadStage-3] | 2017-08-28 16:06:44.118000 | 172.31.32.220 | 6151 | 172.31.16.220
Read 1 live and 0 tombstone cells [ReadStage-3] | 2017-08-28 16:06:44.118000 | 172.31.32.220 | 6203 | 172.31.16.220
Enqueuing response to /172.31.16.220 [ReadStage-3] | 2017-08-28 16:06:44.118001 | 172.31.32.220 | 6228 | 172.31.16.220
REQUEST_RESPONSE message received from /172.31.32.220 [MessagingService-Incoming-/172.31.32.220] | 2017-08-28 16:06:44.119000 | 172.31.16.220 | 9891 | 172.31.16.220
Sending REQUEST_RESPONSE message to cassandra-test-kenobi.aws.local/172.31.16.220 [MessagingService-Outgoing-cassandra-test-kenobi.aws.local/172.31.16.220-Small] | 2017-08-28 16:06:44.119000 | 172.31.32.220 | 7466 | 172.31.16.220
Processing response from /172.31.32.220 [RequestResponseStage-3] | 2017-08-28 16:06:44.120000 | 172.31.16.220 | 10013 | 172.31.16.220
Request complete | 2017-08-28 16:06:44.124753 | 172.31.16.220 | 14753 | 172.31.16.220