Java Hadoop:将多个IP地址绑定到群集名称节点
我在Softlayer上有一个四节点Hadoop集群。主机(NameNode)有一个用于外部访问的公共IP地址和一个用于集群访问的私有IP地址。从节点(datanodes)具有私有IP地址,我正在尝试将其连接到主节点,而无需为每个从节点分配公共IP地址 我意识到,将Java Hadoop:将多个IP地址绑定到群集名称节点,java,hadoop,network-programming,cluster-computing,distributed-computing,Java,Hadoop,Network Programming,Cluster Computing,Distributed Computing,我在Softlayer上有一个四节点Hadoop集群。主机(NameNode)有一个用于外部访问的公共IP地址和一个用于集群访问的私有IP地址。从节点(datanodes)具有私有IP地址,我正在尝试将其连接到主节点,而无需为每个从节点分配公共IP地址 我意识到,将fs.defaultFS设置为NameNode的公共地址允许外部访问,除了NameNode只监听传入连接的地址,而不监听私有地址。因此,我在datanode日志中获得ConnectionUndered异常,因为它们试图连接NameNo
fs.defaultFS
设置为NameNode的公共地址允许外部访问,除了NameNode只监听传入连接的地址,而不监听私有地址。因此,我在datanode日志中获得ConnectionUndered异常,因为它们试图连接NameNode的私有IP地址
我认为解决方案可能是将公共和私有IP地址都设置为NameNode,以便保留外部访问,并允许从属节点连接
那么,有没有一种方法可以将这两个地址绑定到NameNode,以便它同时侦听这两个地址
编辑:Hadoop 2.4.1版。提问者将此编辑成他的问题作为回答: 在hdfs-site.xml中,将
dfs.namenode.rpc bind host
的值设置为
0.0.0.0
和Hadoop将在私有和公共网络上监听
允许远程访问和数据节点访问的网络接口
而且是在地面上完成的
注意:在开始修改之前,请按如下方式停止代理和服务器
如下:
hdfs site.xml
文件后,启动
名称节点上的服务器和代理以及上的代理
数据节点(这不会
使用以下方法(如果也这样做了)损坏群集:
HDFS配置中
有一个名为Bind NameNode to Wildcard Address
的属性,只需选中该框,它就会将服务绑定到0.0.0.0
然后重新启动hdfs服务
您可以发布自己的编辑作为答案-然后我可以投票it@ikradex您好,感谢您提供的解决方案,但有一点我仍然存在问题,即节点管理器仍然无法连接到资源管理器,即使我将“Thread.nodemanager.bind host”的值设置为0.0.0。对此有任何建议???@StarLord请避免使用评论提出(后续)问题。如果你有问题,你可以问一个新问题。当然,你可能想参考这个答案,但一定要确保问题本身写得正确。
<property>
<name>dfs.namenode.rpc-bind-host</name>
<value>0.0.0.0</value>
<description>
The actual address the RPC server will bind to. If this optional address is
set, it overrides only the hostname portion of dfs.namenode.rpc-address.
It can also be specified per name node or name service for HA/Federation.
This is useful for making the name node listen on all interfaces by
setting it to 0.0.0.0.
</description>
</property>
<property>
<name>dfs.namenode.rpc-bind-host</name>
<value>0.0.0.0</value>
<description>
The actual address the RPC server will bind to. If this optional address is
set, it overrides only the hostname portion of dfs.namenode.rpc-address.
It can also be specified per name node or name service for HA/Federation.
This is useful for making the name node listen on all interfaces by
setting it to 0.0.0.0.
</description>
</property>
<property>
<name>dfs.namenode.servicerpc-bind-host</name>
<value>0.0.0.0</value>
<description>
The actual address the service RPC server will bind to. If this optional address is
set, it overrides only the hostname portion of dfs.namenode.servicerpc-address.
It can also be specified per name node or name service for HA/Federation.
This is useful for making the name node listen on all interfaces by
setting it to 0.0.0.0.
</description>
</property>
<property>
<name>dfs.namenode.http-bind-host</name>
<value>0.0.0.0</value>
<description>
The actual adress the HTTP server will bind to. If this optional address
is set, it overrides only the hostname portion of dfs.namenode.http-address.
It can also be specified per name node or name service for HA/Federation.
This is useful for making the name node HTTP server listen on all
interfaces by setting it to 0.0.0.0.
</description>
</property>
<property>
<name>dfs.namenode.https-bind-host</name>
<value>0.0.0.0</value>
<description>
The actual adress the HTTPS server will bind to. If this optional address
is set, it overrides only the hostname portion of dfs.namenode.https-address.
It can also be specified per name node or name service for HA/Federation.
This is useful for making the name node HTTPS server listen on all
interfaces by setting it to 0.0.0.0.
</description>
</property>
To configure HDFS to bind to all the interfaces , add the following configuration variable using Ambari under the section HDFS
-> Configs ->Advanced -> Custom hdfs-site
dfs.namenode.rpc-bind-host = 0.0.0.0
Restart HDFS to apply the configuration change .
Verify if port 8020 is bound and listening to requests from all the interfaces using the following command.
netstat -anp|grep 8020
tcp 0 0 0.0.0.0:8020 0.0.0.0:* LISTEN 15826/java
On the Home > Status tab, click to the right of the service
name and select Restart. Click Start on the next screen to confirm.
When you see a Finished status, the service has restarted.