Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/apache-kafka/3.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Java MapReduce Avro输出正在创建文本文件_Java_Hadoop_Mapreduce_Avro - Fatal编程技术网

Java MapReduce Avro输出正在创建文本文件

Java MapReduce Avro输出正在创建文本文件,java,hadoop,mapreduce,avro,Java,Hadoop,Mapreduce,Avro,我有一个MapReduce任务,读取avro数据,然后输出avro数据。但是,当我在作业成功时检查输出文件时,它们没有.avro扩展名,我可以使用简单的文本编辑器查看它们 我有我的驱动程序配置为输出avro,所以我不知道是什么问题,任何帮助将不胜感激 这是我的驾驶课: public class Driver extends Configured implements Tool{ public static void main(String[] args) throws Exception {

我有一个MapReduce任务,读取avro数据,然后输出avro数据。但是,当我在作业成功时检查输出文件时,它们没有.avro扩展名,我可以使用简单的文本编辑器查看它们

我有我的驱动程序配置为输出avro,所以我不知道是什么问题,任何帮助将不胜感激

这是我的驾驶课:

public class Driver extends Configured implements Tool{

public static void main(String[] args) throws Exception {
    int res =
            ToolRunner.run(new Configuration(), new Driver(), args);
    System.exit(res);
}

@Override
public int run(String[] args) throws Exception {
    Job job = new Job(getConf());
    job.setJarByClass(Driver.class);
    job.setJobName("nearestpatient");


    AvroJob.setOutputKeySchema(job, Pair.getPairSchema(Schema.create(Schema.Type.LONG), Schema.create(Schema.Type.STRING)));
    job.setOutputValueClass(NullWritable.class);

    job.setMapperClass(PatientMapper.class);
    job.setReducerClass(PatientReducer.class);

    job.setInputFormatClass(AvroKeyInputFormat.class);
    AvroJob.setInputKeySchema(job, PatientAvro.getClassSchema());

    job.setMapOutputKeyClass(LongWritable.class);
    job.setMapOutputValueClass(LongWritable.class);


    FileInputFormat.setInputPaths(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    job.waitForCompletion(true);

    return 0;
}
}
这是我的减速机课程:

public class PatientReducer extends Reducer<LongWritable, LongWritable, AvroWrapper<Pair<Long, String>>, NullWritable> {

    @Override
    public void reduce(LongWritable providerKey, Iterable<LongWritable> patients, Context context) throws IOException, InterruptedException {

        String outputList = "[";
   `enter code here` List<Long> patientList = new ArrayList<>();
    for (LongWritable patientKey : patients) {
        outputList += new LongWritable(patientKey.get()) + ", ";
    }
    outputList = outputList.substring(0, outputList.length() - 2);
    outputList += "]";
    context.write(new AvroWrapper<Pair<Long, String>>(new Pair<Long, String> (providerKey.get(), outputList)), NullWritable.get());
}
}
公共类PatientDucer扩展了Reducer{
@凌驾
public void reduce(LongWritable providerKey、Iterable patients、Context Context)抛出IOException、interruptedeexception{
字符串outputList=“[”;
`在此处输入代码`List patientList=new ArrayList();
for(可长写的patientKey:patients){
outputList+=新的LongWritable(patientKey.get())+“,”;
}
outputList=outputList.substring(0,outputList.length()-2);
输出列表+=“]”;
write(新的AvroWrapper(新的对(providerKey.get(),outputList)),nullwriteable.get());
}
}
在run()方法中,需要添加以下内容

job.setOutputFormatClass(AvroKeyValueOutputFormat.class);
在run()方法中,需要添加以下内容

job.setOutputFormatClass(AvroKeyValueOutputFormat.class);

在代码替换行中

FileOutputFormat.setOutputPath(job, new Path(args[1]));


在代码替换行中

FileOutputFormat.setOutputPath(job, new Path(args[1]));