Cassandra 卡桑德拉-压实卡滞

预先警告-卡桑德拉初学者

我使用datastax enterprise ami在aws上设置了一个4节点的m3.xlarge集群，并使用Cassandra bulkloader方法加载数据

卡桑德拉版本是“发布版本：2.1.9.791”

四个节点中的一个——我开始加载Bukload的节点——似乎处于压缩状态（在过去12个小时内没有任何变化）

我还注意到，有时该节点变得不可用（在opscenter中变为红色），但过了一段时间（很长一段时间）它又变为可用

在cassandra日志中有一个例外（见下文）。但奇怪的是，还有很多磁盘空间

> ERROR [MemtableFlushWriter:3] 2015-10-29 23:54:21,511 
> CassandraDaemon.java:223 - Exception in thread
> Thread[MemtableFlushWriter:3,5,main]
> org.apache.cassandra.io.FSWriteError: java.io.IOException: No space
> left on device
>         at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:663)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
>         at org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:500)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
>         at org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:453)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
>         at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:445)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
>         at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:440)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
>         at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:389)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
>         at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:335)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
>         at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.1.jar:na]
>         at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1154)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_80]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_80]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80] Caused by: java.io.IOException: No space left on device
>         at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.7.0_80]
>         at java.io.FileOutputStream.write(FileOutputStream.java:345) ~[na:1.7.0_80]
>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[na:1.7.0_80]
>         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[na:1.7.0_80]
>         at org.apache.cassandra.io.util.DataOutputStreamPlus.flush(DataOutputStreamPlus.java:55)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
>         at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:657)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
>         ... 12 common frames omitted

Tpstats输出为

   $ nodetool tpstats
Pool Name                    Active   Pending      Completed   Blocked  All time blocked
CounterMutationStage              0         0              0         0                 0
ReadStage                         0         0          19485         0                 0
RequestResponseStage              0         0         116191         0                 0
MutationStage                     0         0         386132         0                 0
ReadRepairStage                   0         0            848         0                 0
GossipStage                       0         0          46669         0                 0
CacheCleanupExecutor              0         0              0         0                 0
AntiEntropyStage                  0         0              0         0                 0
MigrationStage                    0         0              1         0                 0
Sampler                           0         0              0         0                 0
ValidationExecutor                0         0              0         0                 0
CommitLogArchiver                 0         0              0         0                 0
MiscStage                         0         0              0         0                 0
MemtableFlushWriter               0         0             80         0                 0
MemtableReclaimMemory             0         0             79         0                 0
PendingRangeCalculator            0         0              4         0                 0
MemtablePostFlush                 1        33            127         0                 0
CompactionExecutor                1         1          27492         0                 0
InternalResponseStage             0         0              4         0                 0
HintedHandoff                     0         0              3         0                 0

Message type           Dropped
RANGE_SLICE                  0
READ_REPAIR                  0
PAGED_RANGE                  0
BINARY                       0
READ                         0
MUTATION                     0
_TRACE                       0
REQUEST_RESPONSE             0
COUNTER_MUTATION             0

有没有人能给我们一些建议，告诉我们如何消除这种现象，以及为什么会发生这种情况

非常感谢所有提示

德克萨斯州

---

Peter

假设您使用的是SizeTieredCompression，并且您有四个大小为X的SSTable，一次压缩会将它们合并为一个大小为Y的SSTable，并且此过程会自动重复

问题：压实将创建一个新的大小为Y的表，两个新的和旧的X大小的表都存在于压实过程中

在最坏的情况下（没有删除和覆盖），压缩将需要2倍用于SSTABLE的磁盘空间，或者更具体地说：在某些情况下，您需要有足够的磁盘空间来容纳大小为X和Y的SSTABLE

因此，即使看起来您还有足够的空间，在压缩过程中也可能会耗尽磁盘空间

您可能想尝试LeveledCompactionStrategy，因为它需要的压缩空间要少得多（10 x表大小，单位为mb）。另请参阅，了解何时使用LeveledCompactionStrategy

---

无论您使用哪种压缩策略，都应始终留出足够的可用磁盘空间来容纳流式处理、修复和快照。

谢谢！清理剩余的散装货物并重新启动。压实再次开始。希望现在一切顺利！

   $ nodetool tpstats
Pool Name                    Active   Pending      Completed   Blocked  All time blocked
CounterMutationStage              0         0              0         0                 0
ReadStage                         0         0          19485         0                 0
RequestResponseStage              0         0         116191         0                 0
MutationStage                     0         0         386132         0                 0
ReadRepairStage                   0         0            848         0                 0
GossipStage                       0         0          46669         0                 0
CacheCleanupExecutor              0         0              0         0                 0
AntiEntropyStage                  0         0              0         0                 0
MigrationStage                    0         0              1         0                 0
Sampler                           0         0              0         0                 0
ValidationExecutor                0         0              0         0                 0
CommitLogArchiver                 0         0              0         0                 0
MiscStage                         0         0              0         0                 0
MemtableFlushWriter               0         0             80         0                 0
MemtableReclaimMemory             0         0             79         0                 0
PendingRangeCalculator            0         0              4         0                 0
MemtablePostFlush                 1        33            127         0                 0
CompactionExecutor                1         1          27492         0                 0
InternalResponseStage             0         0              4         0                 0
HintedHandoff                     0         0              3         0                 0

Message type           Dropped
RANGE_SLICE                  0
READ_REPAIR                  0
PAGED_RANGE                  0
BINARY                       0
READ                         0
MUTATION                     0
_TRACE                       0
REQUEST_RESPONSE             0
COUNTER_MUTATION             0