Python 卡桑德拉崩溃了，不知道出了什么问题

系统在一段时间后崩溃，但出现以下异常：

Python-Pycassa异常读取：

错误：尝试连接到每个服务器两次， 但没有一次尝试成功。最后一次失败是 t传输异常：无法连接到10.242.253.141:9160

卡桑德拉系统日志

 WARN [MemoryMeter:1] 2012-07-31 12:44:17,555 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.16881217597040166
 INFO [MemoryMeter:1] 2012-07-31 12:44:17,555 Memtable.java (line 213) CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is 1.0 (just-counted was 1.0).  calculation took 28ms for 122 columns
 WARN [MemoryMeter:1] 2012-07-31 12:44:22,583 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.11368422493368459
 INFO [MemoryMeter:1] 2012-07-31 12:44:22,586 Memtable.java (line 213) CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is 1.0 (just-counted was 1.0).  calculation took 39ms for 358 columns
 INFO [GossipTasks:1] 2012-07-31 12:44:22,609 Gossiper.java (line 811) InetAddress /10.204.81.77 is now dead.
 WARN [MemoryMeter:1] 2012-07-31 12:44:25,244 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.13325571841646536
 INFO [MemoryMeter:1] 2012-07-31 12:44:25,244 Memtable.java (line 213) CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is 1.0 (just-counted was 1.0).  calculation took 46ms for 484 columns
 INFO [StorageServiceShutdownHook] 2012-07-31 12:44:26,679 CassandraDaemon.java (line 218) Stop listening to thrift clients
 INFO [StorageServiceShutdownHook] 2012-07-31 12:44:26,762 MessagingService.java (line 539) Waiting for messaging service to quiesce
 INFO [ACCEPT-/10.242.253.141] 2012-07-31 12:44:26,780 MessagingService.java (line 695) MessagingService shutting down server thread.
ERROR [Thrift:40765] 2012-07-31 12:44:26,858 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
    at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
    at org.apache.cassandra.service.StorageProxy.scheduleLocalHint(StorageProxy.java:386)
    at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:339)
    at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:119)
    at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:260)
    at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:193)
    at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:637)
    at org.apache.cassandra.thrift.CassandraServer.internal_insert(CassandraServer.java:511)
    at org.apache.cassandra.thrift.CassandraServer.insert(CassandraServer.java:519)
    at org.apache.cassandra.thrift.Cassandra$Processor$insert.getResult(Cassandra.java:3016)
    at org.apache.cassandra.thrift.Cassandra$Processor$insert.getResult(Cassandra.java:3004)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
ERROR [Thrift:42739] 2012-07-31 12:44:26,935 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
    at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
    at org.apache.cassandra.service.StorageProxy.scheduleLocalHint(StorageProxy.java:386)
    at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:339)
    at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:119)
    at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:260)
    at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:193)
    at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:637)
    at org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:587)
    at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:595)
    at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3112)
    at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3100)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

cassanda.yaml

cluster_name: "Test Cluster"

initial_token:

hinted_handoff_enabled: true
max_hint_window_in_ms: 3600000 # one hour
hinted_handoff_throttle_delay_in_ms: 1

authenticator: org.apache.cassandra.auth.AllowAllAuthenticator

authority: org.apache.cassandra.auth.AllowAllAuthority

partitioner: org.apache.cassandra.dht.RandomPartitioner

data_file_directories:
    - /mnt
    - /ebs

commitlog_directory: /mnt/cassandra_log/

key_cache_size_in_mb:

key_cache_save_period: 14400


row_cache_size_in_mb: 0

row_cache_save_period: 0


row_cache_provider: SerializingCacheProvider

saved_caches_directory: /var/lib/cassandra/saved_caches

commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000

seed_provider:
    # Addresses of hosts that are deemed contact points. 
    # Cassandra nodes use this list of hosts to find each other and learn
    # the topology of the ring.  You must change this if you are running
    # multiple nodes!
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          # seeds is actually a comma-delimited list of addresses.
          # Ex: "<ip1>,<ip2>,<ip3>"
          - seeds: "127.0.0.1"

flush_largest_memtables_at: 0.45

reduce_cache_sizes_at: 0.45
reduce_cache_capacity_to: 0.45

concurrent_reads: 8
concurrent_writes: 8

memtable_total_space_in_mb: 64

commitlog_total_space_in_mb: 64

memtable_flush_writers: 1

memtable_flush_queue_size: 1

trickle_fsync: false
trickle_fsync_interval_in_kb: 10240

storage_port: 7776

ssl_storage_port: 7001

listen_address: 10.242.253.141


rpc_address: 0.0.0.0
rpc_port: 9160

rpc_keepalive: true

rpc_server_type: sync



thrift_framed_transport_size_in_mb: 15

thrift_max_message_length_in_mb: 16

incremental_backups: false

snapshot_before_compaction: false

auto_snapshot: true

column_index_size_in_kb: 64

in_memory_compaction_limit_in_mb: 64

multithreaded_compaction: false

compaction_throughput_mb_per_sec: 16

compaction_preheat_key_cache: true

rpc_timeout_in_ms: 10000

endpoint_snitch: SimpleSnitch

dynamic_snitch_update_interval_in_ms: 100 
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1

request_scheduler: org.apache.cassandra.scheduler.NoScheduler

index_interval: 128

encryption_options:
    internode_encryption: none
    keystore: conf/.keystore
    keystore_password: cassandra
    truststore: conf/.truststore
    truststore_password: cassandra
    # More advanced defaults below:
    # protocol: TLS
    # algorithm: SunX509
    # store_type: JKS
    # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA]

cluster\u名称：“测试集群”
初始令牌：
提示切换已启用：true
最大提示窗口（单位：3600000）一小时
ms中的提示切换限制延迟：1
验证者：org.apache.cassandra.auth.AllowAllAuthenticator
授权：org.apache.cassandra.auth.AllowAllAuthority
分区器：org.apache.cassandra.dht.RandomPartitioner
数据文件目录：
-/mnt
-/ebs
提交日志目录：/mnt/cassandra\u日志/
密钥缓存大小（单位：mb）：
密钥缓存保存周期：14400
行缓存大小（单位：mb）：0
行缓存保存周期：0
行缓存提供程序：序列化缓存提供程序
保存的\u缓存\u目录：/var/lib/cassandra/saved\u缓存
commitlog_同步：定期
提交日志同步周期（毫秒）：10000
种子提供程序：
#被视为联系人的主机地址。
#Cassandra节点使用此主机列表来相互查找和学习
#环的拓扑结构。如果正在运行，则必须更改此选项
#多个节点！
-类名：org.apache.cassandra.locator.SimpleSeedProvider
参数：
#种子实际上是一个以逗号分隔的地址列表。
#例：“，”
-种子：“127.0.0.1”
最大冲水高度为：0.45
将缓存大小减少到0.45
将缓存容量减少到：0.45
同时读：8
并发写入：8
memtable_总空间（单位：mb）：64
commitlog_总空间（单位：mb）：64
memtable_flush_写入程序：1
内存表\u刷新\u队列\u大小：1
涓流同步：错误
涓流同步间隔（单位：kb）：10240
存储端口：7776
ssl_存储_端口：7001
收听地址：10.242.253.141
rpc_地址：0.0.0.0
rpc_端口：9160
rpc_keepalive：正确
rpc_服务器_类型：同步
节约型框架运输型尺寸单位：15
节俭\u最大消息长度\u mb:16
增量备份：false
压缩前的快照：false
自动快照：true
列索引大小（kb）：64
内存中的压缩限制内存中的大小：64
多线程压缩：false
每秒压缩量：16
压缩\u预热\u键\u缓存：真
rpc_超时（单位：毫秒）：10000
端点_告密者：SimpleSnitch
动态\u飞贼\u更新\u间隔\u毫秒：100
动态飞贼重置间隔时间：600000毫秒
动态窃贼不良阈值：0.1
请求调度程序：org.apache.cassandra.scheduler.NoScheduler
索引间隔：128
加密选项：
节间加密：无
密钥库：conf/.keystore
密钥库密码：cassandra
信任库：conf/.truststore
信任库密码：cassandra
#更高级的默认设置如下：
#协议：TLS
#算法：SunX509
#门店类型：JKS
#密码套件：[TLS\U RSA\U带AES\U 128\U CBC\U SHA，TLS\U RSA\U带AES\U 256\CBC\U SHA]

---

那么出了什么问题？请引导？

看起来Cassandra进程正在关闭，可能是因为发送了一个信号。这些线条表明：

INFO [StorageServiceShutdownHook] 2012-07-31 12:44:26,679 CassandraDaemon.java (line 218) Stop listening to thrift clients
INFO [StorageServiceShutdownHook] 2012-07-31 12:44:26,762 MessagingService.java (line 539) Waiting for messaging service to quiesce
INFO [ACCEPT-/10.242.253.141] 2012-07-31 12:44:26,780 MessagingService.java (line 695) MessagingService shutting down server thread.
ERROR [Thrift:40765] 2012-07-31 12:44:26,858 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down

---

我也有同样的问题。我首先想到两件事

sudo vim /etc/cassandra/cassandra.yaml

将侦听ip更改为您在AWS上的私有ip，并将rpc_ip更改为0.0.0.0

还要确保您的服务器至少有2GB RAM，否则cassandra将继续崩溃

要在ubuntu上启动cassandra，请尝试以下命令：

sudo service cassandra restart

并且要检查状态吗

sudo service cassandra status

---

希望这有助于确保您没有以某种方式向流程发送术语信号。除此之外，它可能是linux OOM的杀手。不管它是什么，它可能是Cassandra本身的外部。我检查过了，我没有发送这个信号，linux会因为内存/cpu限制而发送吗？是的，如果每个进程都有任何类型的资源限制，很可能就是这样。