Java 从mapper将2个矩阵作为键和值发出时的问题
我得到一个输入文件作为参数,并将其复制到HDFS中进行一些初始计算Java 从mapper将2个矩阵作为键和值发出时的问题,java,hadoop,mapreduce,mapper,writable,Java,Hadoop,Mapreduce,Mapper,Writable,我得到一个输入文件作为参数,并将其复制到HDFS中进行一些初始计算 这个文件将作为我的映射器的输入 在一些矩阵计算之后,我需要向Reducer发送两个双数组。 但当我这么做的时候,它表明: 13/10/26 09:13:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 13/10/2
这个文件将作为我的映射器的输入
在一些矩阵计算之后,我需要向Reducer发送两个双数组。 但当我这么做的时候,它表明:
13/10/26 09:13:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/10/26 09:13:49 WARN conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id
13/10/26 09:13:49 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
13/10/26 09:13:49 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/10/26 09:13:49 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
13/10/26 09:13:49 INFO input.FileInputFormat: Total input paths to process : 1
13/10/26 09:13:50 INFO mapred.LocalJobRunner: OutputCommitter set in config null
13/10/26 09:13:50 INFO mapred.JobClient: Running job: job_local1799392614_0001
13/10/26 09:13:50 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
13/10/26 09:13:50 INFO mapred.LocalJobRunner: Waiting for map tasks
13/10/26 09:13:50 INFO mapred.LocalJobRunner: Starting task: attempt_local1799392614_0001_m_000000_0
13/10/26 09:13:50 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
13/10/26 09:13:50 INFO util.ProcessTree: setsid exited with exit code 0
13/10/26 09:13:50 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@40cdc794
13/10/26 09:13:50 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
13/10/26 09:13:50 INFO mapred.MapTask: io.sort.mb = 100
13/10/26 09:13:50 INFO mapred.MapTask: data buffer = 79691776/99614720
13/10/26 09:13:50 INFO mapred.MapTask: record buffer = 262144/327680
13/10/26 09:13:50 INFO mapred.LocalJobRunner: Map task executor complete.
13/10/26 09:13:50 WARN mapred.LocalJobRunner: job_local1799392614_0001
java.lang.Exception: java.lang.ClassCastException: class edu.Driver$DoubleArrayWritable
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
Caused by: java.lang.ClassCastException: class edu.Driver$DoubleArrayWritable
at java.lang.Class.asSubclass(Class.java:3037)
at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:819)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:376)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:85)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:584)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:656)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
13/10/26 09:13:51 INFO mapred.JobClient: map 0% reduce 0%
13/10/26 09:13:51 INFO mapred.JobClient: Job complete: job_local1799392614_0001
13/10/26 09:13:51 INFO mapred.JobClient: Counters: 0
我的2d阵列是:
public static class DoubleArrayWritable extends TwoDArrayWritable {
public DoubleArrayWritable() {
super(DoubleWritable.class);
}
}
我的Mapper类:
public class Mapper extends Mapper<Object, Text, DoubleArrayWritable, DoubleArrayWritable> {
public void map(Object key, Text value, Context context)
throws IOException, InterruptedException {
DoubleArrayWritable EArray = new DoubleArrayWritable();
DoubleWritable[][] Edata = new DoubleWritable[Erow][Ecol];
for (int k = 0; k < Erow; k++) {
for(int j=0;j< Ecol;j++){
Edata[k][j] = new DoubleWritable(E[k][j]);
}
}
EArray.set(Edata);
DoubleArrayWritable EDArray = new DoubleArrayWritable();
DoubleWritable[][] EDdata = new DoubleWritable[EtransDerow][EtransDecol];
for (int k1 = 0; k1 < EtransDerow; k1++) {
for(int j1=0;j1< EtransDecol;j1++) {
EDdata[k1][j1] = new DoubleWritable(ED[k1][j1]);
}
}
EDArray.set(EDdata);
context.write(EArray, EDArray);
}
}
制图员:
public class Mapper extends
Mapper<Object, Text, IntWritable, DoubleArrayWritable> {
public void map(Object key, Text value, Context context)
throws IOException, InterruptedException {
IntWritable clusterNumber = null;
int count = 0;
clusterNumber = new IntWritable(count);
context.write(clusterNumber, EArray);
}
}
公共类映射器扩展
制图员{
公共无效映射(对象键、文本值、上下文)
抛出IOException、InterruptedException{
IntWritable clusterNumber=null;
整数计数=0;
clusterNumber=新的可写整数(计数);
context.write(clusterNumber,array);
}
}
减速器:
public class Reducer extends
Reducer<IntWritable, DoubleArrayWritable, IntWritable, Text> {
public void reduce(IntWritable key,
Iterable<DoubleArrayWritable> values, Context context)
throws IOException, InterruptedException {
System.out.println("Entered into reducer. successfull");
}
}
公共类缩减器扩展
减速器{
public void reduce(可写密钥,
Iterable值,上下文)
抛出IOException、InterruptedException{
System.out.println(“已成功输入减速机”);
}
}
其次,我尝试发射两个双数组。通过更改配置,映射输出键/值和减速机输出键/值。
似乎
配置中有一些问题
看起来像是类加载器问题..我正在用3个单独的文件编写代码驱动程序、映射程序、还原程序。我还提到了驱动程序类中的类。那么,为什么会出现例外呢
job.setJarByClass(Driver.class);
job.setMapperClass(Mapper.class);
job.setReducerClass(Reducer.class);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(DoubleArrayWritable.class);
job.setOutputKeyClass(TextOutputFormat.class);
job.setOutputValueClass(DoubleArrayWritable.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.setInputPaths(job, "in/inputfile");
FileOutputFormat.setOutputPath(job,new Path(args[1]));
public class Mapper extends
Mapper<Object, Text, IntWritable, DoubleArrayWritable> {
public void map(Object key, Text value, Context context)
throws IOException, InterruptedException {
IntWritable clusterNumber = null;
int count = 0;
clusterNumber = new IntWritable(count);
context.write(clusterNumber, EArray);
}
}
public class Reducer extends
Reducer<IntWritable, DoubleArrayWritable, IntWritable, Text> {
public void reduce(IntWritable key,
Iterable<DoubleArrayWritable> values, Context context)
throws IOException, InterruptedException {
System.out.println("Entered into reducer. successfull");
}
}