Java/Scala远程HDFS使用_Java_Scala_Hadoop_Hdfs

Java/Scala远程HDFS使用

java scala hadoop

Java/Scala远程HDFS使用,java,scala,hadoop,hdfs,Java,Scala,Hadoop,Hdfs,我正在尝试连接到远程HDFS群集。我已经阅读了一些文档和入门指南，但没有找到最好的解决方案。情境：我在xxx-something.com上有HDFS。我可以通过SSH连接到它，一切正常但我要做的是，把文件从它拿到我的本地机器上我所做的：我已经在conf文件夹中创建了core-site.xml（我正在创建Play！应用程序）。在这里，我将fs.default.name配置更改为hdfs://xxx-something.com:8020 （不确定端口）。然后我尝试启动一个简单的测试： va

我正在尝试连接到远程HDFS群集。我已经阅读了一些文档和入门指南，但没有找到最好的解决方案。情境：我在xxx-something.com上有HDFS。我可以通过SSH连接到它，一切正常

但我要做的是，把文件从它拿到我的本地机器上

我所做的：

我已经在conf文件夹中创建了core-site.xml（我正在创建Play！应用程序）。在这里，我将fs.default.name配置更改为hdfs://xxx-something.com:8020 （不确定端口）。然后我尝试启动一个简单的测试：

val conf = new Configuration()
conf.addResource(new Path("conf/core-site.xml"))
val fs = FileSystem.get(conf)
val status = fs.listStatus(new Path("/data/"))

我发现了一些错误：

 13:56:09.012 [specs2.DefaultExecutionStrategy1] WARN  org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: fs.trash.interval;  Ignoring.
13:56:09.012 [specs2.DefaultExecutionStrategy1] WARN  org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: hadoop.tmp.dir;  Ignoring.
13:56:09.013 [specs2.DefaultExecutionStrategy1] WARN  org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: fs.checkpoint.dir;  Ignoring.
13:56:09.022 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.fs.FileSystem - Creating filesystem for hdfs://xxx-something.com:8021
13:56:09.059 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.conf.Configuration - java.io.IOException: config()
    at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:226)
    at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:213)
    at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:53)
    at org.apache.hadoop.net.NetUtils.<clinit>(NetUtils.java:62)

然后：

Call to xxx-something.com/someIp:22 failed on local exception: java.io.EOFException
java.io.IOException: Call to xxx-something.com/someIp:22 failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
    at org.apache.hadoop.ipc.Client.call(Client.java:1071)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:122)
    at HdfsSpec$$anonfun$1$$anonfun$apply$3.apply(HdfsSpec.scala:33)
    at HdfsSpec$$anonfun$1$$anonfun$apply$3.apply(HdfsSpec.scala:17)
    at testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.executeBody(MyNotifierRunner.scala:16)
    at testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.execute(MyNotifierRunner.scala:16)
Caused by: java.io.EOFException
    at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:807)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:745)

调用xxx something.com/someIp:22失败，本地异常：java.io.EOFException java.io.IOException:调用xxx something.com/someIp:22在本地异常上失败：java.io.eofeexception 位于org.apache.hadoop.ipc.Client.wrapException（Client.java:1103）位于org.apache.hadoop.ipc.Client.call（Client.java:1071）位于org.apache.hadoop.ipc.RPC$Invoker.invoke（RPC.java:225）位于com.sun.proxy.$Proxy1.getProtocolVersion（未知源）位于org.apache.hadoop.ipc.RPC.getProxy（RPC.java:396）位于org.apache.hadoop.ipc.RPC.getProxy（RPC.java:379）位于org.apache.hadoop.hdfs.DFSClient.createRPCNamenode（DFSClient.java:118）位于org.apache.hadoop.hdfs.DFSClient.（DFSClient.java:222）位于org.apache.hadoop.hdfs.DFSClient.（DFSClient.java:187）位于org.apache.hadoop.hdfs.DistributedFileSystem.initialize（DistributedFileSystem.java:89）位于org.apache.hadoop.fs.FileSystem.createFileSystem（FileSystem.java:1328）位于org.apache.hadoop.fs.FileSystem.access$200（FileSystem.java:65）位于org.apache.hadoop.fs.FileSystem$Cache.get（FileSystem.java:1346）位于org.apache.hadoop.fs.FileSystem.get（FileSystem.java:244）位于org.apache.hadoop.fs.FileSystem.get（FileSystem.java:122）在HdfsSpec$$anonfun$1$$anonfun$apply$3.apply处（HdfsSpec.scala:33）在HdfsSpec$$anonfun$1$$anonfun$apply$3.apply处（HdfsSpec.scala:17）在testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.executeBody（MyNotifierRunner.scala:16）上在testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.execute（MyNotifierRunner.scala:16）原因：java.io.EOFException 位于org.apache.hadoop.ipc.Client$Connection.receiveResponse（Client.java:807）位于org.apache.hadoop.ipc.Client$Connection.run（Client.java:745）

这意味着什么？

您需要在运行name节点（HDFS master）的服务器上的$HADOOP_HOME/conf/core-site.xml中找到fs.default.name属性，才能获得正确的端口。可能是8020，也可能是别的什么。这就是你应该使用的。确保您和服务器之间没有防火墙禁止端口上的连接

Call to xxx-something.com/someIp:22 failed on local exception: java.io.EOFException
java.io.IOException: Call to xxx-something.com/someIp:22 failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
    at org.apache.hadoop.ipc.Client.call(Client.java:1071)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:122)
    at HdfsSpec$$anonfun$1$$anonfun$apply$3.apply(HdfsSpec.scala:33)
    at HdfsSpec$$anonfun$1$$anonfun$apply$3.apply(HdfsSpec.scala:17)
    at testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.executeBody(MyNotifierRunner.scala:16)
    at testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.execute(MyNotifierRunner.scala:16)
Caused by: java.io.EOFException
    at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:807)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:745)