Hadoop 无法识别我的代码中的错误
我有两个数据集:Hadoop 无法识别我的代码中的错误,hadoop,mapreduce,mapper,reducers,Hadoop,Mapreduce,Mapper,Reducers,我有两个数据集: 用户: 用户日志: Sunny NewPlayer 12.23.14.421 Klopp Crazy 88.33.44.555 Bobby NewPlayer 99.12.11.222 Steven Captain 99.55.66.777 Jamie Local 88.99.33.232 Suarez Spain 77.55.66.444 我想使用reducer join连接这两个数据集。 我是这样写我的课程的: MapperClass: Public class Mapp
用户: 用户日志:
Sunny NewPlayer 12.23.14.421
Klopp Crazy 88.33.44.555
Bobby NewPlayer 99.12.11.222
Steven Captain 99.55.66.777
Jamie Local 88.99.33.232
Suarez Spain 77.55.66.444
我想使用reducer join连接这两个数据集。
我是这样写我的课程的:
MapperClass:
Public class MapperClass {
public static class UserMap extends Mapper<LongWritable, Text, Text, Text> {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
String[] tokens = line.split(" ");
String name = tokens[0];
String city = tokens[2];
context.write(new Text(name), new Text("UserFile" + "\t" + city));
}
}
public static class UserLogs extends Mapper<LongWritable, Text, Text, Text> {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
String[] tokens = line.split(" ");
String name = tokens[0];
String ip = tokens[2];
context.write(new Text(name), new Text("UserLogs" + "\t" + ip));
}
}
}
输出应如下所示:
Bobby Amsterdam 99.12.11.222
Sunny Rotterdam 12.23.14.421
Klopp Liverpool 88.33.44.555
Steven Liverpool 99.55.66.777
Jamie Liverpool 88.99.33.232
Suarez Barcelona 77.55.66.444
Bobby Amsterdam IP Address not found
Jamie Liverpool 88.99.33.232
Klopp Liverpool IP Address not found
Macca Liverpool IP Address not found
Messi Barcelona IP Address not found
Neymar brazil IP Address not found
Pique Barcelona IP Address not found
Steven Liverpool 99.55.66.777
Suarez Barcelona IP Address not found
Sunny Rotterdam 12.23.14.421
相反,我得到的输出如下:
Bobby Amsterdam 99.12.11.222
Sunny Rotterdam 12.23.14.421
Klopp Liverpool 88.33.44.555
Steven Liverpool 99.55.66.777
Jamie Liverpool 88.99.33.232
Suarez Barcelona 77.55.66.444
Bobby Amsterdam IP Address not found
Jamie Liverpool 88.99.33.232
Klopp Liverpool IP Address not found
Macca Liverpool IP Address not found
Messi Barcelona IP Address not found
Neymar brazil IP Address not found
Pique Barcelona IP Address not found
Steven Liverpool 99.55.66.777
Suarez Barcelona IP Address not found
Sunny Rotterdam 12.23.14.421
我不明白我在这里犯了什么错误。
有人能帮我解决这个问题吗。
非常感谢您提供的任何帮助。您的reducer中有一个错误,它将根据
值的顺序覆盖IP地址。试试这个:
public class ReducerClass extends Reducer<Text, Text, Text, Text>{
@Override
public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
String city = null;
String ip = null;
for(Text t: values) {
String[] parts = t.toString().split("\t");
if(parts[0].equals("UserFile")) {
city = parts[1];
} else if(parts[0].equals("UserLogs")) {
ip = parts[1];
}
}
if (ip != null && city != null) {
context.write(key, new Text(city + "\t" + ip));
}
}
}
公共类Reducer类扩展Reducer{
@凌驾
公共void reduce(文本键、Iterable值、上下文上下文)引发IOException、InterruptedException{
字符串city=null;
字符串ip=null;
对于(文本t:值){
String[]parts=t.toString().split(“\t”);
if(部分[0].equals(“用户文件”)){
城市=部分[1];
}else if(部分[0].equals(“用户日志”)){
ip=零件[1];
}
}
如果(ip!=null和城市!=null){
编写(键,新文本(城市+“\t”+ip));
}
}
}
您的reducer中有一个错误,它将根据值的顺序覆盖IP地址。试试这个:
public class ReducerClass extends Reducer<Text, Text, Text, Text>{
@Override
public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
String city = null;
String ip = null;
for(Text t: values) {
String[] parts = t.toString().split("\t");
if(parts[0].equals("UserFile")) {
city = parts[1];
} else if(parts[0].equals("UserLogs")) {
ip = parts[1];
}
}
if (ip != null && city != null) {
context.write(key, new Text(city + "\t" + ip));
}
}
}
公共类Reducer类扩展Reducer{
@凌驾
公共void reduce(文本键、Iterable值、上下文上下文)引发IOException、InterruptedException{
字符串city=null;
字符串ip=null;
对于(文本t:值){
String[]parts=t.toString().split(“\t”);
if(部分[0].equals(“用户文件”)){
城市=部分[1];
}else if(部分[0].equals(“用户日志”)){
ip=零件[1];
}
}
如果(ip!=null和城市!=null){
编写(键,新文本(城市+“\t”+ip));
}
}
}
纠正了它,效果很好。非常感谢Mariusz。更正了它,效果很好。非常感谢Mariusz。