Fatal编程技术网
  • cassandra/
  • 无法让DataStax OpsCenter 4.0.3连接到Cassandra 2.0.4群集

无法让DataStax OpsCenter 4.0.3连接到Cassandra 2.0.4群集

cassandra

无法让DataStax OpsCenter 4.0.3连接到Cassandra 2.0.4群集,cassandra,datastax,opscenter,Cassandra,Datastax,Opscenter,我有一个单节点C*2.0.4集群正在运行，nodetool状态显示集群运行正常 然后，我使用“sudo-yum-install-OpsCenter-free”在同一网络上的另一台机器上安装了OpsCenter 4.0.3 在opscenterd.conf文件中，我设置了接口='OpsCenter服务器的公共IP'，并启动了OpsCenter服务器 然后我可以看到OpsCenter网页并单击使用现有集群 在AddCluster接口下，我输入了单节点Cassandra集群的rpc_地址。OpsCen

我有一个单节点C*2.0.4集群正在运行，nodetool状态显示集群运行正常

然后，我使用“sudo-yum-install-OpsCenter-free”在同一网络上的另一台机器上安装了OpsCenter 4.0.3

在opscenterd.conf文件中，我设置了接口='OpsCenter服务器的公共IP'，并启动了OpsCenter服务器

然后我可以看到OpsCenter网页并单击使用现有集群

在AddCluster接口下，我输入了单节点Cassandra集群的rpc_地址。OpsCenter接受了它，并在下一页正确显示了集群名称

但是，OpsCenter中没有一个图形加载，我看到了错误：连接了0个代理中的0个。我还看到一个闪烁的红色X，上面有一个插头图标

在CentOS中，OpsCenter和C*节点上的防火墙当前都已关闭

如何让Opscener正确连接到C*节点

下面是OpsCenter日志显示的内容（注意：我用A.B.C.D替换了IP）：

在Cassandra节点上，一切看起来都很健康：

[root@cassandra01 ~]# nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns   Host ID                               Rack
UN  A.B.C.D  158.27 KB  256     100.0%  bc560cd6-a20d-4b36-99ca-ed477dc939b5  rack1
但是，我无法卷曲OpsCenter试图访问的URL：

[root@cassandra01 ~]# curl http://A.B.C.D:61621/cluster/topology?node_ip=A.B.C.D
curl: (7) couldn't connect to host
在OpsCenterd.conf文件中关闭SSL（默认设置）

以下是我在以下URL中看到的内容http://Public Opscener的IP:8888/狗/节点

[{"load": null, "has_jna": false, "vnodes": true, "devices": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "task_progress": {}, "node_ip": "A.B.C.D", "network_interfaces": null, "ec2": {}, "node_version": {}, "dc": null, "node_name": null, "num_procs": null, "streaming": {}, "token": "5743408169174478324", "data_held": null, "mode": "unknown", "rpc_ip": "10.183.132.141", "partitions": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "os": null, "rack": null, "last_seen": 0}]
有没有办法解决这个问题

注意，在Cassandra的YAML文件中，rpc_服务器_类型设置为sync

更新：

我还尝试使用“yum install datastax agent”在C*节点上手动安装OpsCenter代理，然后使用以下设置编辑address.yaml文件：

stomp_interface: 'public ip of machine opscenterd is running on (public IP)'
local_interface: 'listen_address in cassandra.yaml (public IP)'
agent_rpc_interface: 'rpc address in cassandra.yaml (private IP network)'
agent_rpc_broadcast_address: 'private network IP, same network at rpc address'
我为address.yaml文件尝试了一些不同的设置，但没有一个有效。例如，我试图只设置stop_接口，并删除了其他3行。没用。我还试图设置停止和本地接口，但这也不起作用

当我现在使用“服务datastax代理启动”启动datastax代理时，Cassandra服务突然崩溃：

[root@cassandra01~]#sudo服务卡桑德拉状态 cassandra已死亡，但pid文件存在

当C*服务崩溃时，opscenter代理将保持运行状态。如果我停止代理服务并再次启动C*服务（sudo service cassandra状态），则C*将成功启动，nodetool状态显示一个健康的单节点集群。但是，当我启动代理服务时，C*服务突然再次崩溃。我在address.yaml文件中尝试的所有不同设置都会导致相同的行为

理想情况下，我不想手动安装代理，只想将它的安装程序从OpsCenter GUI推送到C*节点上，但由于这不起作用，我尝试手动安装代理并将其连接到OpsCenter，但不幸的是，这也不起作用

有时，当Cassandra服务崩溃时，我在Cassandra节点上也会看到这一点： [root@cassandra01~]#sudo服务cassandra stoplog4j：警告：找不到记录器（org.eclipse.jetty.util.log）的附件。 log4j：警告请正确初始化log4j系统。 log4j：有关更多信息，请参阅警告。 用法：卡桑德拉启动|停止|状态|重新启动|重新加载

以下是Cassandra节点的log4j-server.properties：

log4j.rootLogger=INFO,stdout,R

# stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p %d{HH:mm:ss,SSS} %m%n

# rolling log file
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.maxFileSize=20MB
log4j.appender.R.maxBackupIndex=50
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%5p [%t] %d{ISO8601} %F (line %L) %m%n
# Edit the next line to point to your logs directory
log4j.appender.R.File=/var/log/cassandra/system.log

# Application logging options
#log4j.logger.org.apache.cassandra=DEBUG
#log4j.logger.org.apache.cassandra.db=DEBUG
#log4j.logger.org.apache.cassandra.service.StorageProxy=DEBUG

# Adding this to avoid thrift logging disconnect errors.
log4j.logger.org.apache.thrift.server.TNonblockingServer=ERROR
最后，下面是运行在Cassandra节点上的OpsCenter代理的agent.log显示的内容：

nohup: ignoring input
Starting DataStax agent monitor datastax_agent_monitor
 INFO [main] 2014-01-30 08:24:59,104 Loading conf files: /var/lib/datastax-agent /conf/address.yaml
 INFO [main] 2014-01-30 08:24:59,261 Java vendor/version: Java HotSpot(TM) 64-Bi t Server VM/1.7.0_25
 INFO [main] 2014-01-30 08:24:59,546 Default config values: {:rollups300_ttl 241 9200, :settings_cf "settings", :agent_rpc_interface "10.183.132.141", :my_channe l_prefix "/agent", :poll_period 60, :kerberos_hostname nil, :storage_dc nil, :th rift_conn_timeout 10000, :thrift_max_frame_size 15728640, :rollups60_ttl 604800,  :stomp_port 61620, :shorttime_interval 10, :longtime_interval 300, :private-con f-props ["initial_token" "listen_address" "broadcast_address" "rpc_address"], :t hrift_port 9160, :async_retry_timeout 5, :agent-conf-group "global-cluster-agent -group", :jmx_host "127.0.0.1", :ec2_metadata_api_host "169.254.169.254", :metri cs_enabled 1, :async_queue_size 5000, :autodiscovery_interval 120, :rollups7200_ ttl 31536000, :autodiscovery_enabled true, :thrift_ssl_truststore nil, :rollup_s napshot_period 300, :is_package true, :monitor_command "/usr/share/datastax-agen t/bin/datastax_agent_monitor", :thrift_socket_timeout 5000, :cassandra_log_locat ion "/var/log/cassandra/system.log", :local_interface "23.253.64.169", :jmx_port  7199, :jmx_metrics_threadpool_size 4, :use_ssl 0, :rollups86400_ttl -1, :nodede tails_threadpool_size 3, :api_port 61621, :kerberos_service nil, :kerberos_clien t_principal nil, :jmx_thread_pool_size 5, :production 1, :stomp_interface "166.7 8.186.184", :storage_keyspace "OpsCenter", :rollup_snapshot_threshold 300, :thri ft_ssl_truststore_type "JKS", :realtime_interval 5}
 INFO [main] 2014-01-30 08:24:59,554 Waiting for the config from OpsCenter
 INFO [main] 2014-01-30 08:24:59,559 Using 23.253.64.169 as the cassandra broadc ast address
 INFO [main] 2014-01-30 08:24:59,568 New JMX connection (127.0.0.1:7199)
ERROR [main] 2014-01-30 08:25:00,019 Error connecting via JMX: java.io.IOExcepti on: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException  [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0. 0.1; nested exception is:
        java.net.ConnectException: Connection refused]
 INFO [main] 2014-01-30 08:25:00,414 cassandra RPC address is  nil
 INFO [main] 2014-01-30 08:25:00,418 agent RPC broadcast address is  10.183.132. 141
 INFO [main] 2014-01-30 08:25:00,474 Clearing ssl.truststore
 INFO [main] 2014-01-30 08:25:00,475 Clearing ssl.truststore.password
 INFO [main] 2014-01-30 08:25:00,476 Setting ssl.store.type to JKS
 INFO [main] 2014-01-30 08:25:00,477 Clearing kerberos.service.principal.name
 INFO [main] 2014-01-30 08:25:00,480 Clearing kerberos.principal
 INFO [main] 2014-01-30 08:25:00,480 Clearing kerberos.useTicketCache
 INFO [main] 2014-01-30 08:25:00,481 Clearing kerberos.ticketCache
 INFO [main] 2014-01-30 08:25:00,487 Clearing kerberos.useKeyTab
 INFO [main] 2014-01-30 08:25:00,487 Clearing kerberos.keyTab
 INFO [main] 2014-01-30 08:25:00,487 Clearing kerberos.renewTGT
 INFO [main] 2014-01-30 08:25:00,488 Clearing kerberos.debug
 INFO [main] 2014-01-30 08:25:00,495 Starting Stomp
 INFO [main] 2014-01-30 08:25:00,495 SSL communication is disabled
 INFO [main] 2014-01-30 08:25:00,495 Creating stomp connection to 166.78.186.184 :61620
 INFO [thrift-init] 2014-01-30 08:25:00,521 Connecting to Cassandra cluster: 23. 253.64.169 (port 9160)
 INFO [StompConnection receiver] 2014-01-30 08:25:00,536 Reconnecting in 0s.
 INFO [StompConnection receiver] 2014-01-30 08:25:00,561 Connected to 166.78.186 .184:61620
 INFO [thrift-init] 2014-01-30 08:25:00,619 Downed Host Retry service started wi th queue size -1 and retry delay 10s
 INFO [thrift-init] 2014-01-30 08:25:00,662 Registering JMX me.prettyprint.cassa ndra.service_Agent Cluster:ServiceType=hector,MonitorType=hector
 INFO [main] 2014-01-30 08:25:00,732 Starting Jetty server: {:port 61621, :host  "10.183.132.141", :ssl? false, :join? false}
ERROR [thrift-init] 2014-01-30 08:25:00,885 MARK HOST AS DOWN TRIGGERED for host  23.253.64.169(23.253.64.169):9160
ERROR [thrift-init] 2014-01-30 08:25:00,886 Pool state on shutdown: <ConcurrentC assandraClientPoolByHost>:{23.253.64.169(23.253.64.169):9160}; IsActive?: true;  Active: 0; Blocked: 1; Idle: 0; NumBeforeExhausted: 1
 INFO [thrift-init] 2014-01-30 08:25:00,887 Shutdown triggered on <ConcurrentCas sandraClientPoolByHost>:{23.253.64.169(23.253.64.169):9160}
 INFO [thrift-init] 2014-01-30 08:25:00,901 Shutdown complete on <ConcurrentCass andraClientPoolByHost>:{23.253.64.169(23.253.64.169):9160}
 INFO [thrift-init] 2014-01-30 08:25:00,902 Host detected as down was added to r etry queue: 23.253.64.169(23.253.64.169):9160
 WARN [thrift-init] 2014-01-30 08:25:00,914 Could not fullfill request on this h ost null
 WARN [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] 2 014-01-30 08:25:00,910 Downed 23.253.64.169(23.253.64.169):9160 host still appea rs to be down: Unable to open transport to 23.253.64.169(23.253.64.169):9160 , j ava.net.ConnectException: Connection refused
 WARN [thrift-init] 2014-01-30 08:25:00,926 Exception:
me.prettyprint.hector.api.exceptions.HectorTransportException: Unable to open tr ansport to 23.253.64.169(23.253.64.169):9160 , java.net.ConnectException: Connec tion refused
        at me.prettyprint.cassandra.connection.client.HThriftClient.open(HThrift Client.java:180)
        at me.prettyprint.cassandra.connection.client.HThriftClient.open(HThrift Client.java:38)
        at me.prettyprint.cassandra.connection.ConcurrentHClientPool.createClien t(ConcurrentHClientPool.java:162)
        at me.prettyprint.cassandra.connection.ConcurrentHClientPool.borrowClien t(ConcurrentHClientPool.java:94)
        at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFai lover(HConnectionManager.java:250)
        at me.prettyprint.cassandra.service.AbstractCluster.describeClusterName( AbstractCluster.java:155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
        at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
        at clj_hector.core$cluster_name.invoke(core.clj:40)
        at opsagent.cassandra$setup_cassandra$f__353__auto____900$fn__920.invoke (cassandra.clj:360)
        at opsagent.cassandra$setup_cassandra$f__353__auto____900.invoke(cassand ra.clj:358)
        at clojure.lang.AFn.run(AFn.java:24)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectExce ption: Connection refused
        at org.apache.thrift.transport.TSocket.open(TSocket.java:183)
        at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.ja va:81)
        at me.prettyprint.cassandra.connection.client.HThriftClient.open(HThrift Client.java:174)
        ... 16 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
        at java.net.SocksSocketImpl.connect(Unknown Source)
        at java.net.Socket.connect(Unknown Source)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:178)
        ... 18 more
ERROR [thrift-init] 2014-01-30 08:25:00,965 Error when performing thrift operati on:
me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down . Retry burden pushed out to client.
        at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromL BPolicy(HConnectionManager.java:395)
        at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFai lover(HConnectionManager.java:249)
        at me.prettyprint.cassandra.service.AbstractCluster.describeClusterName( AbstractCluster.java:155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
        at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
        at clj_hector.core$cluster_name.invoke(core.clj:40)
        at opsagent.cassandra$setup_cassandra$f__353__auto____900$fn__920.invoke (cassandra.clj:360)
        at opsagent.cassandra$setup_cassandra$f__353__auto____900.invoke(cassand ra.clj:358)
        at clojure.lang.AFn.run(AFn.java:24)
        at java.lang.Thread.run(Unknown Source)
 INFO [StompConnection receiver] 2014-01-30 08:25:01,024 Got new config from Ops Center: {:kerberos_use_keytab true, :rollups300_ttl 2419200, :kerberos_use_ticke t_cache true, :rollups60_ttl 604800, :thrift_port 9160, :ec2_metadata_api_host " 169.254.169.254", :metrics_enabled 1, :rollups7200_ttl 31536000, :thrift_ssl_tru ststore nil, :metrics_ignored_column_families "", :cassandra_log_location "/var/ log/cassandra/system.log", :thrift_rpc_interface "10.183.132.141", :thrift_ssl_t ruststore_password nil, :jmx_port 7199, :provisioning 0, :use_ssl 0, :kerberos_d ebug false, :rollups86400_ttl -1, :api_port "61621", :storage_keyspace "OpsCente r", :kerberos_renew_tgt true, :metrics_ignored_solr_cores "", :thrift_ssl_trusts tore_type "JKS", :metrics_ignored_keyspaces "system, system_traces, system_auth,  dse_auth, OpsCenter", :rollup_subscriptions [], :cassandra_install_location ""}
 INFO [StompConnection receiver] 2014-01-30 08:25:01,030 Starting up agent colle ction.
 INFO [StompConnection receiver] 2014-01-30 08:25:01,040 New JMX connection (127 .0.0.1:7199)
ERROR [StompConnection receiver] 2014-01-30 08:25:01,073 Error connecting via JM X: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceU navailableException [Root exception is java.rmi.ConnectException: Connection ref used to host: 127.0.0.1; nested exception is:
        java.net.ConnectException: Connection refused]
 INFO [Jetty] 2014-01-30 08:25:01,160 Jetty server started
 INFO [StompConnection receiver] 2014-01-30 08:25:01,188 Starting OS metric coll ectors (Linux)
 INFO [StompConnection receiver] 2014-01-30 08:25:01,199 Starting Cassandra JMX  metric collectors
 INFO [install-location-finder] 2014-01-30 08:25:01,250 New JMX connection (127. 0.0.1:7199)
 INFO [StompConnection receiver] 2014-01-30 08:25:01,252 New JMX connection (127 .0.0.1:7199)
ERROR [install-location-finder] 2014-01-30 08:25:01,261 Error connecting via JMX : java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUn availableException [Root exception is java.rmi.ConnectException: Connection refu sed to host: 127.0.0.1; nested exception is:
        java.net.ConnectException: Connection refused]

nohup:忽略输入
正在启动DataStax代理监视器DataStax\u代理监视器
INFO[main]2014-01-30 08:24:59104加载conf文件：/var/lib/datastax agent/conf/address.yaml
信息[主要]2014-01-30 08:24:59261 Java供应商/版本：Java HotSpot（TM）64 Bi t服务器VM/1.7.025
信息[主要]2014-01-30 08:24:59546默认配置值：{：rollups300\u ttl 241 9200，：设置\u cf“设置”，：代理\u rpc\u接口“10.183.132.141”，：我的通道前缀“/代理”，：轮询周期60，：kerberos主机名为零，：存储为零，：th rift\u conn\u超时10000，：节俭最大帧大小15728640，：rollups60\u ttl 604800，：stomp\u端口61620，：短时间隔10，：长时间隔300，：私有con f-props[“初始令牌”“侦听地址”“广播地址”“rpc\u地址”]，：t hrift_端口9160，：async_重试_超时5，：agent conf group“global cluster agent-group”，：jmx_host“127.0.0.1”，：ec2_metadata_api_host“169.254.169.254”，：metri cs_enabled 1，：async_queue_size 5000，：autodiscovery_interval 120，：rollup7200_ttl 31536000，：autodiscovery_enabled true，：thrift_ssl_truststore nil，：rollup_s napshot_period 300，：is_package true，：monitor\u命令“/usr/share/datasax代理t/bin/datasax代理/datasax代理\u monitor”，：thrift\u套接字\u超时5000，：cassandra\u log\locat“/var/log/cassandra/system.log”、：local_interface“23.253.64.169”、：jmx_port 7199、：jmx_metrics_threadpool_size 4、：use_ssl 0、：rollups86400_ttl-1、：nodede tails_threadpool_size 3、：api_port 61621、：kerberos_service nil、：kerberos_clien t_principal nil、：kerberos_princient nil、：jmx_thread_池_size 5、：生产1、：stomp_interface”166.7.186“，：存储\u键空间“OpsCenter”，：汇总\u快照\u阈值300，：节俭\u ssl\u信任存储\u类型“JKS”，：实时\u间隔5}”
INFO[main]2014-01-30 08:24:59554正在等待OpsCenter的配置
信息[main]2014-01-30 08:24:59559使用23.253.64.169作为卡桑德拉宽带广播地址
信息[主要]2014-01-30 08:24:59568新JMX连接（127.0.0.1:7199）
错误[main]2014-01-30 08:25:00019通过JMX进行连接时出错：上的java.io.IOExcepti:检索RMIServer存根失败：javax.naming.ServiceUnavailableException[根异常为java.rmi.ConnectException:连接拒绝宿主：127.0.0.1；嵌套异常为：
java.net.ConnectException:连接被拒绝]
信息[main]2014-01-30 08:25:00414卡桑德拉RPC地址为零
信息[主]2014-01-30 08:25:00418代理RPC广播地址为10.183.132。141
信息[主要]2014-01-30 08:25:00474清算ssl.truststore
信息[主要]2014-01-30 08:25:00475清算ssl.truststore.pa

nohup: ignoring input
Starting DataStax agent monitor datastax_agent_monitor
 INFO [main] 2014-01-30 08:24:59,104 Loading conf files: /var/lib/datastax-agent /conf/address.yaml
 INFO [main] 2014-01-30 08:24:59,261 Java vendor/version: Java HotSpot(TM) 64-Bi t Server VM/1.7.0_25
 INFO [main] 2014-01-30 08:24:59,546 Default config values: {:rollups300_ttl 241 9200, :settings_cf "settings", :agent_rpc_interface "10.183.132.141", :my_channe l_prefix "/agent", :poll_period 60, :kerberos_hostname nil, :storage_dc nil, :th rift_conn_timeout 10000, :thrift_max_frame_size 15728640, :rollups60_ttl 604800,  :stomp_port 61620, :shorttime_interval 10, :longtime_interval 300, :private-con f-props ["initial_token" "listen_address" "broadcast_address" "rpc_address"], :t hrift_port 9160, :async_retry_timeout 5, :agent-conf-group "global-cluster-agent -group", :jmx_host "127.0.0.1", :ec2_metadata_api_host "169.254.169.254", :metri cs_enabled 1, :async_queue_size 5000, :autodiscovery_interval 120, :rollups7200_ ttl 31536000, :autodiscovery_enabled true, :thrift_ssl_truststore nil, :rollup_s napshot_period 300, :is_package true, :monitor_command "/usr/share/datastax-agen t/bin/datastax_agent_monitor", :thrift_socket_timeout 5000, :cassandra_log_locat ion "/var/log/cassandra/system.log", :local_interface "23.253.64.169", :jmx_port  7199, :jmx_metrics_threadpool_size 4, :use_ssl 0, :rollups86400_ttl -1, :nodede tails_threadpool_size 3, :api_port 61621, :kerberos_service nil, :kerberos_clien t_principal nil, :jmx_thread_pool_size 5, :production 1, :stomp_interface "166.7 8.186.184", :storage_keyspace "OpsCenter", :rollup_snapshot_threshold 300, :thri ft_ssl_truststore_type "JKS", :realtime_interval 5}
 INFO [main] 2014-01-30 08:24:59,554 Waiting for the config from OpsCenter
 INFO [main] 2014-01-30 08:24:59,559 Using 23.253.64.169 as the cassandra broadc ast address
 INFO [main] 2014-01-30 08:24:59,568 New JMX connection (127.0.0.1:7199)
ERROR [main] 2014-01-30 08:25:00,019 Error connecting via JMX: java.io.IOExcepti on: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException  [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0. 0.1; nested exception is:
        java.net.ConnectException: Connection refused]
 INFO [main] 2014-01-30 08:25:00,414 cassandra RPC address is  nil
 INFO [main] 2014-01-30 08:25:00,418 agent RPC broadcast address is  10.183.132. 141
 INFO [main] 2014-01-30 08:25:00,474 Clearing ssl.truststore
 INFO [main] 2014-01-30 08:25:00,475 Clearing ssl.truststore.password
 INFO [main] 2014-01-30 08:25:00,476 Setting ssl.store.type to JKS
 INFO [main] 2014-01-30 08:25:00,477 Clearing kerberos.service.principal.name
 INFO [main] 2014-01-30 08:25:00,480 Clearing kerberos.principal
 INFO [main] 2014-01-30 08:25:00,480 Clearing kerberos.useTicketCache
 INFO [main] 2014-01-30 08:25:00,481 Clearing kerberos.ticketCache
 INFO [main] 2014-01-30 08:25:00,487 Clearing kerberos.useKeyTab
 INFO [main] 2014-01-30 08:25:00,487 Clearing kerberos.keyTab
 INFO [main] 2014-01-30 08:25:00,487 Clearing kerberos.renewTGT
 INFO [main] 2014-01-30 08:25:00,488 Clearing kerberos.debug
 INFO [main] 2014-01-30 08:25:00,495 Starting Stomp
 INFO [main] 2014-01-30 08:25:00,495 SSL communication is disabled
 INFO [main] 2014-01-30 08:25:00,495 Creating stomp connection to 166.78.186.184 :61620
 INFO [thrift-init] 2014-01-30 08:25:00,521 Connecting to Cassandra cluster: 23. 253.64.169 (port 9160)
 INFO [StompConnection receiver] 2014-01-30 08:25:00,536 Reconnecting in 0s.
 INFO [StompConnection receiver] 2014-01-30 08:25:00,561 Connected to 166.78.186 .184:61620
 INFO [thrift-init] 2014-01-30 08:25:00,619 Downed Host Retry service started wi th queue size -1 and retry delay 10s
 INFO [thrift-init] 2014-01-30 08:25:00,662 Registering JMX me.prettyprint.cassa ndra.service_Agent Cluster:ServiceType=hector,MonitorType=hector
 INFO [main] 2014-01-30 08:25:00,732 Starting Jetty server: {:port 61621, :host  "10.183.132.141", :ssl? false, :join? false}
ERROR [thrift-init] 2014-01-30 08:25:00,885 MARK HOST AS DOWN TRIGGERED for host  23.253.64.169(23.253.64.169):9160
ERROR [thrift-init] 2014-01-30 08:25:00,886 Pool state on shutdown: <ConcurrentC assandraClientPoolByHost>:{23.253.64.169(23.253.64.169):9160}; IsActive?: true;  Active: 0; Blocked: 1; Idle: 0; NumBeforeExhausted: 1
 INFO [thrift-init] 2014-01-30 08:25:00,887 Shutdown triggered on <ConcurrentCas sandraClientPoolByHost>:{23.253.64.169(23.253.64.169):9160}
 INFO [thrift-init] 2014-01-30 08:25:00,901 Shutdown complete on <ConcurrentCass andraClientPoolByHost>:{23.253.64.169(23.253.64.169):9160}
 INFO [thrift-init] 2014-01-30 08:25:00,902 Host detected as down was added to r etry queue: 23.253.64.169(23.253.64.169):9160
 WARN [thrift-init] 2014-01-30 08:25:00,914 Could not fullfill request on this h ost null
 WARN [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] 2 014-01-30 08:25:00,910 Downed 23.253.64.169(23.253.64.169):9160 host still appea rs to be down: Unable to open transport to 23.253.64.169(23.253.64.169):9160 , j ava.net.ConnectException: Connection refused
 WARN [thrift-init] 2014-01-30 08:25:00,926 Exception:
me.prettyprint.hector.api.exceptions.HectorTransportException: Unable to open tr ansport to 23.253.64.169(23.253.64.169):9160 , java.net.ConnectException: Connec tion refused
        at me.prettyprint.cassandra.connection.client.HThriftClient.open(HThrift Client.java:180)
        at me.prettyprint.cassandra.connection.client.HThriftClient.open(HThrift Client.java:38)
        at me.prettyprint.cassandra.connection.ConcurrentHClientPool.createClien t(ConcurrentHClientPool.java:162)
        at me.prettyprint.cassandra.connection.ConcurrentHClientPool.borrowClien t(ConcurrentHClientPool.java:94)
        at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFai lover(HConnectionManager.java:250)
        at me.prettyprint.cassandra.service.AbstractCluster.describeClusterName( AbstractCluster.java:155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
        at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
        at clj_hector.core$cluster_name.invoke(core.clj:40)
        at opsagent.cassandra$setup_cassandra$f__353__auto____900$fn__920.invoke (cassandra.clj:360)
        at opsagent.cassandra$setup_cassandra$f__353__auto____900.invoke(cassand ra.clj:358)
        at clojure.lang.AFn.run(AFn.java:24)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectExce ption: Connection refused
        at org.apache.thrift.transport.TSocket.open(TSocket.java:183)
        at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.ja va:81)
        at me.prettyprint.cassandra.connection.client.HThriftClient.open(HThrift Client.java:174)
        ... 16 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
        at java.net.SocksSocketImpl.connect(Unknown Source)
        at java.net.Socket.connect(Unknown Source)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:178)
        ... 18 more
ERROR [thrift-init] 2014-01-30 08:25:00,965 Error when performing thrift operati on:
me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down . Retry burden pushed out to client.
        at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromL BPolicy(HConnectionManager.java:395)
        at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFai lover(HConnectionManager.java:249)
        at me.prettyprint.cassandra.service.AbstractCluster.describeClusterName( AbstractCluster.java:155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
        at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
        at clj_hector.core$cluster_name.invoke(core.clj:40)
        at opsagent.cassandra$setup_cassandra$f__353__auto____900$fn__920.invoke (cassandra.clj:360)
        at opsagent.cassandra$setup_cassandra$f__353__auto____900.invoke(cassand ra.clj:358)
        at clojure.lang.AFn.run(AFn.java:24)
        at java.lang.Thread.run(Unknown Source)
 INFO [StompConnection receiver] 2014-01-30 08:25:01,024 Got new config from Ops Center: {:kerberos_use_keytab true, :rollups300_ttl 2419200, :kerberos_use_ticke t_cache true, :rollups60_ttl 604800, :thrift_port 9160, :ec2_metadata_api_host " 169.254.169.254", :metrics_enabled 1, :rollups7200_ttl 31536000, :thrift_ssl_tru ststore nil, :metrics_ignored_column_families "", :cassandra_log_location "/var/ log/cassandra/system.log", :thrift_rpc_interface "10.183.132.141", :thrift_ssl_t ruststore_password nil, :jmx_port 7199, :provisioning 0, :use_ssl 0, :kerberos_d ebug false, :rollups86400_ttl -1, :api_port "61621", :storage_keyspace "OpsCente r", :kerberos_renew_tgt true, :metrics_ignored_solr_cores "", :thrift_ssl_trusts tore_type "JKS", :metrics_ignored_keyspaces "system, system_traces, system_auth,  dse_auth, OpsCenter", :rollup_subscriptions [], :cassandra_install_location ""}
 INFO [StompConnection receiver] 2014-01-30 08:25:01,030 Starting up agent colle ction.
 INFO [StompConnection receiver] 2014-01-30 08:25:01,040 New JMX connection (127 .0.0.1:7199)
ERROR [StompConnection receiver] 2014-01-30 08:25:01,073 Error connecting via JM X: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceU navailableException [Root exception is java.rmi.ConnectException: Connection ref used to host: 127.0.0.1; nested exception is:
        java.net.ConnectException: Connection refused]
 INFO [Jetty] 2014-01-30 08:25:01,160 Jetty server started
 INFO [StompConnection receiver] 2014-01-30 08:25:01,188 Starting OS metric coll ectors (Linux)
 INFO [StompConnection receiver] 2014-01-30 08:25:01,199 Starting Cassandra JMX  metric collectors
 INFO [install-location-finder] 2014-01-30 08:25:01,250 New JMX connection (127. 0.0.1:7199)
 INFO [StompConnection receiver] 2014-01-30 08:25:01,252 New JMX connection (127 .0.0.1:7199)
ERROR [install-location-finder] 2014-01-30 08:25:01,261 Error connecting via JMX : java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUn availableException [Root exception is java.rmi.ConnectException: Connection refu sed to host: 127.0.0.1; nested exception is:
        java.net.ConnectException: Connection refused]



Jan 30 20:06:39 hostname kernel: Out of memory: Kill process 2900 (java) score 788 or sacrifice child
Jan 30 20:06:39 hostname kernel: Killed process 2900, UID 0, (java) total-vm:1383360kB, anon-rss:717176kB, file-rss:113316kB



system_memory_in_mb=`free -m | awk '/Mem:/ {print $2}'`



system_memory_in_mb=`free -m | awk '/Mem:/ {print $4}'`