Java Hdfs显示本地文件的列表

我在OSX上安装了Hadoop，一切都很顺利。我的经验是最近的，正在努力学习更多关于Hadoop应用程序开发的知识

昨天，当我需要在Hadoop中查找目录和/或文件列表时，我只需键入

$ hadoop fs -ls 

它会显示集群中的所有内容

现在，它显示了文件系统中的所有本地内容。我必须提供hdfs的确切地址才能得到目录

$ hadoop fs -ls hdfs://localhost:8020/user/myName

我的

core site.xml

文件与以前一样

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<!-- Put site-specific property overrides in this file. -->
 <configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:8020</value>
    </property>
</configuration>

我得到管理报告，通知文件系统文件：///不是HDFS文件系统

$ hadoop dfsadmin -report
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.

2018-10-18 18:01:27,316 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
report: FileSystem file:/// is not an HDFS file system
Usage: hdfs dfsadmin [-report] [-live] [-dead] [-decommissioning] [-enteringmaintenance] [-inmaintenance]

在

core site.xml

文件中，我还将配置更新为：

<property>
    <!-- <name>fs.default.name</name> -->
    <!-- <value>hdfs://localhost:8020</value> -->
    <name>fs.defaultFS</name>
    <value>hdfs://localhost.localdomain:8020/</value>
</property>

---

如何切换到HDFS文件系统？任何形式的建议都将不胜感激

在core-site.xml文件中进行如下编辑

<value>hdfs://localhost.localdomain:8020/</value>

hdfs://localhost.localdomain:8020/

我相信遗漏斜杠（8020/）会造成问题。

---

试试看

您需要确保添加了一个名为

HADOOP\u CONF\u DIR

的环境变量，以设置为包含来自HADOOP的XML文件的目录

您可以在主文件夹的

.bashrc

中执行此操作

---

否则，您将得到默认的文件系统，

file://

，该文件系统仍然有效，对于运行MapReduce作业仍然可以正常工作

---

FWIW，这是我的核心网站

$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:///tmp/hadoop/hdfs/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

$cat/usr/local/ceral/hadoop/3.1.1/libexec/etc/hadoop/core-site.xml
hadoop.tmp.dir
file:///tmp/hadoop/hdfs/tmp
其他临时目录的基础。
fs.default.name
hdfs://localhost:9000

和hdfs站点

$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
  <property>
      <name>dfs.namenode.name.dir</name>
      <value>file:///tmp/hadoop/hdfs/names</value>
  </property>
  <property>
    <name>fs.checkpoint.dir</name>
    <value>file:///tmp/hadoop/hdfs/checkpoint</value>
  </property>
  <property>
    <name>fs.checkpoint.edits.dir</name>
    <value>file:///tmp/hadoop/hdfs/checkpoint-edits</value>
  </property>
  <property>
      <name>dfs.datanode.data.dir</name>
      <value>file:///tmp/hadoop/hdfs/data</value>
  </property>
</configuration>

$cat/usr/local/ceral/hadoop/3.1.1/libexec/etc/hadoop/hdfs-site.xml
dfs.replication
1.
dfs.namenode.name.dir
file:///tmp/hadoop/hdfs/names
fs.checkpoint.dir
file:///tmp/hadoop/hdfs/checkpoint
fs.checkpoint.edits.dir
file:///tmp/hadoop/hdfs/checkpoint-edits
dfs.datanode.data.dir
file:///tmp/hadoop/hdfs/data

我们在Cloudera数据平台7.1.5边缘节点（没有主节点也没有从节点：只有客户端和Cloudera管理器）上遇到了相同的问题。HDFS文件通常显示在每个集群节点上，但显示本地文件系统的边缘节点除外。解决方案是在我们的边缘节点上安装网关角色，如所述（感谢链接），

不起作用。我相信这是另外一回事，因为它使用的是昨天在

XML

文件中编写的相同配置。请删除注释行并重试。还是不走运？看看这个博客，因为它处理的是同一个问题：评论的行不编译。。。静止的，无效的。稍后我将检查您的链接。我现在试着写一个MR算法。我不能接受你的回答。。对不起，如果它不能解决问题，那么你就不应该接受答案：/

$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:///tmp/hadoop/hdfs/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
  <property>
      <name>dfs.namenode.name.dir</name>
      <value>file:///tmp/hadoop/hdfs/names</value>
  </property>
  <property>
    <name>fs.checkpoint.dir</name>
    <value>file:///tmp/hadoop/hdfs/checkpoint</value>
  </property>
  <property>
    <name>fs.checkpoint.edits.dir</name>
    <value>file:///tmp/hadoop/hdfs/checkpoint-edits</value>
  </property>
  <property>
      <name>dfs.datanode.data.dir</name>
      <value>file:///tmp/hadoop/hdfs/data</value>
  </property>
</configuration>