Hadoop NameNode地址的URI无效,s3a不是架构';hdfs';
我正在做一些关于在hadoop环境(纱线)中用Ceph替换HDFS的事情,根据我的研究,来自和的指南显示我需要在Hadoop NameNode地址的URI无效,s3a不是架构';hdfs';,hadoop,hdfs,bigdata,ceph,Hadoop,Hdfs,Bigdata,Ceph,我正在做一些关于在hadoop环境(纱线)中用Ceph替换HDFS的事情,根据我的研究,来自和的指南显示我需要在$hadoop\u home/etc/hadoop下修改core site.xml 我的修改如下: <property> <name>fs.s3a.access.key</name> <value>xxxxxxxxxxxxxx</value> </property> <prop
$hadoop\u home/etc/hadoop
下修改core site.xml
我的修改如下:
<property>
<name>fs.s3a.access.key</name>
<value>xxxxxxxxxxxxxx</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value>xxxxxxxxxxxxx</value>
</property>
<property>
<name>fs.default.name</name>
<value>s3a://bucket_name</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>s3a://bucket_name</value>
</property>
<property>
<name>fs.s3a.endpoint</name>
<value>http://x.x.x.x:xxxx</value>
</property>
<property>
<name>fs.AbstractFileSystem.s3a.imp</name>
<value>org.apache.hadoop.fs.s3a.S3A</value>
</property>
仅供参考,我的hadoop版本是3.2.0
提前谢谢你的帮助 在深入研究了
hadoop
源代码之后,我认为应该抛出这个异常
当您试图调用sbin/start all.sh
时,无法跳过下面的代码
/**
* @return address of file system
*/
public static InetSocketAddress getNNAddress(URI filesystemURI) {
String authority = filesystemURI.getAuthority();
if (authority == null) {
throw new IllegalArgumentException(String.format(
"Invalid URI for NameNode address (check %s): %s has no authority.",
FileSystem.FS_DEFAULT_NAME_KEY, filesystemURI.toString()));
}
if (!HdfsConstants.HDFS_URI_SCHEME.equalsIgnoreCase(
filesystemURI.getScheme())) {
throw new IllegalArgumentException(String.format(
"Invalid URI for NameNode address (check %s): " +
"%s is not of scheme '%s'.", FileSystem.FS_DEFAULT_NAME_KEY,
filesystemURI.toString(), HdfsConstants.HDFS_URI_SCHEME));
}
return getNNAddress(authority);
}
我不需要启动namenode
和secondarynamenode
,因为我使用ceph作为后端存储系统。ceph本身可以通过其驱动程序管理其所谓的datanode
为可能有同样担忧的任何人保留这条线索,并欢迎对我的理解发表任何评论
/**
* @return address of file system
*/
public static InetSocketAddress getNNAddress(URI filesystemURI) {
String authority = filesystemURI.getAuthority();
if (authority == null) {
throw new IllegalArgumentException(String.format(
"Invalid URI for NameNode address (check %s): %s has no authority.",
FileSystem.FS_DEFAULT_NAME_KEY, filesystemURI.toString()));
}
if (!HdfsConstants.HDFS_URI_SCHEME.equalsIgnoreCase(
filesystemURI.getScheme())) {
throw new IllegalArgumentException(String.format(
"Invalid URI for NameNode address (check %s): " +
"%s is not of scheme '%s'.", FileSystem.FS_DEFAULT_NAME_KEY,
filesystemURI.toString(), HdfsConstants.HDFS_URI_SCHEME));
}
return getNNAddress(authority);
}