Java-Hadoop Map Reduce,输入错误,省略csv头
我是Hadoop和MapReduce的新手,通过实现这个程序,现在出现了一系列我不理解的错误 此程序使用具有以下结构的数据集:Barrios.csvJava-Hadoop Map Reduce,输入错误,省略csv头,java,hadoop,mapreduce,Java,Hadoop,Mapreduce,我是Hadoop和MapReduce的新手,通过实现这个程序,现在出现了一系列我不理解的错误 此程序使用具有以下结构的数据集:Barrios.csv "Codigo de barrio";"Codigo de distrito al que pertenece";"Nombre de barrio";"Nombre acentuado del barrio";"Superficie (m2)";"Perimetro (m)" "01";"01";"PALACIO ";"P
"Codigo de barrio";"Codigo de distrito al que pertenece";"Nombre de barrio";"Nombre acentuado del barrio";"Superficie (m2)";"Perimetro (m)"
"01";"01";"PALACIO ";"PALACIO ";"001471085";"005754"
"01";"02";"IMPERIAL ";"IMPERIAL ";"000967500";"004557"
"01";"03";"PACIFICO ";"PACÍFICO ";"000750065";"004005"
"01";"04";"RECOLETOS ";"RECOLETOS ";"000870857";"003927"
"01";"05";"EL VISO ";"EL VISO ";"001708046";"005269"
"01";"06";"BELLAS VISTAS ";"BELLAS VISTAS ";"000716261";"003443"
"01";"07";"GAZTAMBIDE ";"GAZTAMBIDE ";"000506596";"002969"
"01";"08";"EL PARDO ";"EL PARDO ";"187642916";"087125"
"01";"09";"CASA DE CAMPO ";"CASA DE CAMPO ";"017470075";"019233"
"01";"10";"LOS CARMENES ";"LOS CÁRMENES ";"001292235";"006186"
"01";"11";"COMILLAS ";"COMILLAS ";"000665999";"004257"
"01";"12";"ORCASITAS ";"ORCASITAS ";"001356371";"004664"
"01";"13";"ENTREVIAS ";"ENTREVÍAS ";"005996932";"011057"
"01";"14";"PAVONES ";"PAVONES ";"001016979";"004134"
"01";"15";"VENTAS ";"VENTAS ";"003198045";"008207"
"01";"16";"PALOMAS ";"PALOMAS ";"001128602";"004988"
"01";"17";"SAN ANDRES ";"SAN ANDRÉS ";"009192451";"013710"
"01";"18";"CASCO H.VALLECAS ";"CASCO H.VALLECAS ";"049359337";"031924"
"01";"19";"CASCO H.VICALVARO ";"CASCO H.VICÁLVARO ";"032924620";"033326"
"01";"20";"SIMANCAS ";"SIMANCAS ";"002278418";"006678"
"01";"21";"ALAMEDA DE OSUNA ";"ALAMEDA DE OSUNA ";"001961904";"006043"
这代表了马德里的不同地区,并显示了它们的一系列数据,如周长、总表面。。。等
在我的MapReduce程序中,我希望获得按“Codigo de barrio”分组的所有地区的promedium周长,例如,从所有“Codigo de barrio”等于1、然后等于2的地区获得promedium周长。。。etc(oerimeter是最后一列值
这是我的代码:
public class WordCount {
private static final String SEPARATOR = ";";
public static class BarrioMapper extends Mapper<Object, Text, IntWritable, IntWritable>{
public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
final String[] values = value.toString().split(SEPARATOR);
final int grupoBarrio = Integer.parseInt(values[0]);
final int perimetro = Integer.parseInt(values[5]);
context.write(new IntWritable(grupoBarrio), new IntWritable(perimetro));
}
}
public static class BarrioReducer extends Reducer<IntWritable,IntWritable,IntWritable,IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(IntWritable key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int sum = 0;
int contador = 0;
for (IntWritable value : values) {
sum += value.get();
contador++;
}
if (contador > 0) {
result.set(sum/contador);
context.write(key, result);
}
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
job.setJarByClass(WordCount.class);
job.setMapperClass(BarrioMapper.class);
job.setCombinerClass(BarrioReducer.class);
job.setReducerClass(BarrioReducer.class);
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
这个错误是关于“Codigo de barrio”输入数据的错误,我不明白它是什么意思。
`您的拆分错误:
final int grupoBarrio = Integer.parseInt(values[0]);
第一行的值[0]是“Codigo de barrio”,您应该在csv文件中使用ommit头(第一行)-它不是数值,正如错误所说,您不能将头解析为整数 为了跳过该值,可以使用try-catch
try {
final int grupoBarrio = Integer.parseInt(values[0]);
final int perimetro = Integer.parseInt(values[5]);
context.write(new IntWritable(grupoBarrio), new IntWritable(perimetro));
} (NumberFormatException e) { }
或者你应该用没有标题的文件覆盖HDFS文件。你知道如何省略它吗?我认为最好的选择应该是阅读这个问题:并在web中查看map reduce如何工作的许多示例;例如,你可以用不同的方式处理csv标题:手动删除它;尝试/catch parseInt方法;检查文本是否有效ue包含特定标签;阅读此问题:我认为这应该可以解决您的问题:)@fiticida下载文件,删除标题,然后在HDFS中覆盖它
try {
final int grupoBarrio = Integer.parseInt(values[0]);
final int perimetro = Integer.parseInt(values[5]);
context.write(new IntWritable(grupoBarrio), new IntWritable(perimetro));
} (NumberFormatException e) { }