Hadoop 备用NN RPC延迟问题
我目前面临的一个问题是,在我的secondary NameNode上不时会出现RPC延迟问题。该实例上的日志事件如下所示:Hadoop 备用NN RPC延迟问题,hadoop,hadoop2,Hadoop,Hadoop2,我目前面临的一个问题是,在我的secondary NameNode上不时会出现RPC延迟问题。该实例上的日志事件如下所示: The health test result for NAME_NODE_RPC_LATENCY has become bad: The moving average of the RPC latency is 6 second(s) over the previous 5 minute(s). The moving average of the queue time is
The health test result for NAME_NODE_RPC_LATENCY has become bad: The moving average of the RPC latency is 6 second(s) over the previous 5 minute(s). The moving average of the queue time is 0 second(s). The moving average of the processing time is 6 second(s). Critical threshold: 5 second(s).
Time: Sep 25, 2015 5:52:02 AM
我们不时会看到这些RPC错误。我看了日志,没有发现任何不同
我检查了问题发生时的日志,没有发现任何异常
Call#0 Retry#0: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby
这通常是因为客户端不知道连接到何处,并且由于该节点处于备用状态,它会无缝连接到活动NN
我检查了RPC平均队列时间和处理时间,有一次我看到连接中出现突发,我们收到了警报,但另一次情况不好时,请求中没有突发
有什么建议吗?还有什么我可以检查的吗?Call#0重试#0:org.apache.hadoop.ipc.StandbyException:由于存在错误,待机状态下不支持读取操作类别。
Call#0 Retry#0: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby is due to a BUG.
https://issues.apache.org/jira/browse/AMBARI-13373
on similar lines.
If you have HA enabled:
Datanodes are trying to connect to standby namenode because you might have the standby namenode in your dfs.namenode.rpc-address check hdfs-site.xml.
Workaround: Remove this property because dfs.namenode.rpc-address.DEMOMASTER.nn1 and dfs.namenode.rpc-address.DEMOMASTER.nn1 will serve the purpose of dfs.namenode.rpc-address
How to remove?
Use the configs.sh utility on the Ambari Server to delete the extra property.
/var/lib/ambari-server/resources/scripts/configs.sh -u
<admin.user> -p
<admin.password> delete
<ambari.server>
<cluster.name> hdfs-site “dfs.namenode.rpc-address”
Where
admin.user and
admin.password are credentials for an Ambari Administrator,
ambari.server is the Ambari Server host and
cluster.name is the name of your cluster.
https://issues.apache.org/jira/browse/AMBARI-13373
在类似的线路上。
如果已启用HA:
Datanodes正在尝试连接到备用namenode,因为您的dfs.namenode.rpc-address检查hdfs-site.xml中可能有备用namenode。
解决方法:删除此属性,因为dfs.namenode.rpc-address.DEMOMASTER.nn1和dfs.namenode.rpc-address.DEMOMASTER.nn1将用于dfs.namenode.rpc-address
如何删除?
使用Ambari服务器上的configs.sh实用程序删除额外属性。
/var/lib/ambari服务器/resources/scripts/configs.sh-u
-p
删除
hdfs站点“dfs.namenode.rpc地址”
哪里
管理员和用户
admin.password是Ambari管理员的凭据,
ambari.server是ambari服务器主机和
cluster.name是群集的名称。
谢谢您的回复,是的,我们已经在集群上启用了HA。我会调查的。此外,在备用名称节点上发生此延迟的原因可能是什么?