Hadoop mapreduce作业没有';我不会被处决
我有一个简单的MapReduce作业,我从中借用了它,并做了一些小的修改(我删除了reducer)。它基本上以一个简单的avro文件作为输入。下面是avro文件的模式 avro模式:Hadoop mapreduce作业没有';我不会被处决,hadoop,mapreduce,avro,Hadoop,Mapreduce,Avro,我有一个简单的MapReduce作业,我从中借用了它,并做了一些小的修改(我删除了reducer)。它基本上以一个简单的avro文件作为输入。下面是avro文件的模式 avro模式: { "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": "int"}, {"name":"favorite_color", "t
{
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": "int"},
{"name":"favorite_color", "type": "string"}
]
}
下面是我的mapreduce工作(映射器和主要功能):
当我运行程序时,它返回false并且没有成功。我想不出这个问题。有人能帮忙吗?您已将映射器的值类型设置为空可写。然后在main/driver中,将Map输出值类设置为intwriteable。映射器和主/驱动程序中的值类型应该相同。相应地修改您的程序。如果您有解决方案,请接受我的回答。您有解决方案吗?
public class ColorCountMapper extends Mapper<AvroKey<User>, NullWritable, Text, IntWritable> {
@Override
public void map(AvroKey<User> key, NullWritable value, Context context) throws IOException, InterruptedException {
CharSequence color = key.datum().getFavoriteColor();
if (color == null) {
color = "none";
}
context.write(new Text(color.toString()), new IntWritable(1));
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "TestColor");
job.setJarByClass(runClass.class);
job.setJobName("Color Count");
FileInputFormat.setInputPaths(job, new Path("in"));
FileOutputFormat.setOutputPath(job, new Path("out"));
job.setInputFormatClass(AvroKeyInputFormat.class);
job.setMapperClass(ColorCountMapper.class);
AvroJob.setInputKeySchema(job, User.getClassSchema());
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
boolean r = job.waitForCompletion(true);
System.out.println(r);
}