Java MapReduce Avro输出正在创建文本文件
我有一个MapReduce任务,读取avro数据,然后输出avro数据。但是,当我在作业成功时检查输出文件时,它们没有.avro扩展名,我可以使用简单的文本编辑器查看它们 我有我的驱动程序配置为输出avro,所以我不知道是什么问题,任何帮助将不胜感激 这是我的驾驶课:Java MapReduce Avro输出正在创建文本文件,java,hadoop,mapreduce,avro,Java,Hadoop,Mapreduce,Avro,我有一个MapReduce任务,读取avro数据,然后输出avro数据。但是,当我在作业成功时检查输出文件时,它们没有.avro扩展名,我可以使用简单的文本编辑器查看它们 我有我的驱动程序配置为输出avro,所以我不知道是什么问题,任何帮助将不胜感激 这是我的驾驶课: public class Driver extends Configured implements Tool{ public static void main(String[] args) throws Exception {
public class Driver extends Configured implements Tool{
public static void main(String[] args) throws Exception {
int res =
ToolRunner.run(new Configuration(), new Driver(), args);
System.exit(res);
}
@Override
public int run(String[] args) throws Exception {
Job job = new Job(getConf());
job.setJarByClass(Driver.class);
job.setJobName("nearestpatient");
AvroJob.setOutputKeySchema(job, Pair.getPairSchema(Schema.create(Schema.Type.LONG), Schema.create(Schema.Type.STRING)));
job.setOutputValueClass(NullWritable.class);
job.setMapperClass(PatientMapper.class);
job.setReducerClass(PatientReducer.class);
job.setInputFormatClass(AvroKeyInputFormat.class);
AvroJob.setInputKeySchema(job, PatientAvro.getClassSchema());
job.setMapOutputKeyClass(LongWritable.class);
job.setMapOutputValueClass(LongWritable.class);
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
return 0;
}
}
这是我的减速机课程:
public class PatientReducer extends Reducer<LongWritable, LongWritable, AvroWrapper<Pair<Long, String>>, NullWritable> {
@Override
public void reduce(LongWritable providerKey, Iterable<LongWritable> patients, Context context) throws IOException, InterruptedException {
String outputList = "[";
`enter code here` List<Long> patientList = new ArrayList<>();
for (LongWritable patientKey : patients) {
outputList += new LongWritable(patientKey.get()) + ", ";
}
outputList = outputList.substring(0, outputList.length() - 2);
outputList += "]";
context.write(new AvroWrapper<Pair<Long, String>>(new Pair<Long, String> (providerKey.get(), outputList)), NullWritable.get());
}
}
公共类PatientDucer扩展了Reducer{
@凌驾
public void reduce(LongWritable providerKey、Iterable patients、Context Context)抛出IOException、interruptedeexception{
字符串outputList=“[”;
`在此处输入代码`List patientList=new ArrayList();
for(可长写的patientKey:patients){
outputList+=新的LongWritable(patientKey.get())+“,”;
}
outputList=outputList.substring(0,outputList.length()-2);
输出列表+=“]”;
write(新的AvroWrapper(新的对(providerKey.get(),outputList)),nullwriteable.get());
}
}
在run()方法中,需要添加以下内容
job.setOutputFormatClass(AvroKeyValueOutputFormat.class);
在run()方法中,需要添加以下内容
job.setOutputFormatClass(AvroKeyValueOutputFormat.class);
在代码替换行中
FileOutputFormat.setOutputPath(job, new Path(args[1]));
与
在代码替换行中
FileOutputFormat.setOutputPath(job, new Path(args[1]));
与