Warning: file_get_contents(/data/phpspider/zhask/data//catemap/0/hadoop/6.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181

Warning: file_get_contents(/data/phpspider/zhask/data//catemap/4/macos/9.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Java 在hadoop中,如何通过initialize方法初始化DistributedFileSystem对象?_Java_Hadoop - Fatal编程技术网

Java 在hadoop中,如何通过initialize方法初始化DistributedFileSystem对象?

Java 在hadoop中,如何通过initialize方法初始化DistributedFileSystem对象?,java,hadoop,Java,Hadoop,有两个参数,一个URI和一个配置。我假设客户端设置为的JobConf对象应该用于配置,但是URI呢 以下是我为驾驶员提供的代码: JobClient client = new JobClient(); JobConf conf = new JobConf(ClickViewSessions.class); conf.setJobName("ClickViewSessions"); conf.setOutputKeyClass(LongWritable.class); conf.setOutp

有两个参数,一个URI和一个配置。我假设客户端设置为的JobConf对象应该用于配置,但是URI呢

以下是我为驾驶员提供的代码:

JobClient client = new JobClient();
JobConf conf = new JobConf(ClickViewSessions.class);

conf.setJobName("ClickViewSessions");

conf.setOutputKeyClass(LongWritable.class);
conf.setOutputValueClass(MinMaxWritable.class);

FileInputFormat.addInputPath(conf, new Path("input"));
FileOutputFormat.setOutputPath(conf, new Path("output"));

conf.setMapperClass(ClickViewSessionsMapper.class);
conf.setReducerClass(ClickViewSessionsReducer.class);

client.setConf(conf);

DistributedFileSystem dfs = new DistributedFileSystem();
try {
    dfs.initialize(new URI("blah") /* what goes here??? */, conf);
} catch (Exception e) {
    throw new RuntimeException(e.toString());
}

如何获取URI以提供给上面的
初始化
调用?

URI是您正在运行的HDFS的位置。文件系统名称的默认值应为conf/core-site.xml。“fs.default.name”的值应该是您连接到的URI

如果您还没有看过关于如何设置简单单节点系统的教程,我强烈推荐:

您还可以使用如下所示初始化文件系统
导入java.io.IOException;
导入org.apache.hadoop.conf.Configuration;
导入org.apache.hadoop.fs.FileStatus;
导入org.apache.hadoop.fs.FileSystem;
导入org.apache.hadoop.fs.Path;
公共静态void main(字符串参数[]){
试一试{
Configuration conf=新配置();
conf.set(“fs.defaultFS”hdfs://localhost:54310/user/hadoop/");
FileSystem fs=FileSystem.get(conf);
FileStatus[]status=fs.listStatus(新路径(“.”);

对于(inti=0;iDo),您是否运行本地或远程Hadoop集群?
You could also use as shown below to intialize a file system

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

    public static void main(String args[]){
        try {
            Configuration conf = new Configuration();
            conf.set("fs.defaultFS", "hdfs://localhost:54310/user/hadoop/");
            FileSystem fs = FileSystem.get(conf);
            FileStatus[] status = fs.listStatus(new Path("."));
            for(int i=0;i<status.length;i++){
                System.out.println(status[i].getPath());
            }
        } catch (IOException e) {
            e.printStackTrace();
        }

    }