Hadoop 使用MRUnit进行ORC映射器单元测试
我有一个处理ORC文件的地图程序。从驱动程序中,我将ORC格式设置为输入格式Hadoop 使用MRUnit进行ORC映射器单元测试,hadoop,mapreduce,hadoop2,mrunit,orc,Hadoop,Mapreduce,Hadoop2,Mrunit,Orc,我有一个处理ORC文件的地图程序。从驱动程序中,我将ORC格式设置为输入格式 job.setInputFormatClass(OrcNewInputFormat.class); 在OrcNewInputFormat中,值为OrcStruct。在映射方法中,可写的值作为参数(value param)传递,它的类型转换为映射内的OrcStruct,如下所示 OrcStruct record = (OrcStruct) value 我想用MRUnit测试这个映射器。为此,在单元测试的设置方法中,
job.setInputFormatClass(OrcNewInputFormat.class);
在OrcNewInputFormat中,值为OrcStruct。在映射方法中,可写的值作为参数(value param)传递,它的类型转换为映射内的OrcStruct,如下所示
OrcStruct record = (OrcStruct) value
我想用MRUnit测试这个映射器。为此,在单元测试的设置方法中,我在testFilePath中创建了一个ORC文件
OrcFile.createWriter(testFilePath, OrcFile.writerOptions(conf).inspector(inspector).stripeSize(100000).bufferSize(10000).version(OrcFile.Version.V_0_12));
writer.addRow(new SimpleStruct("k1", "v1")) ;
public static class SimpleStruct {
Text k;
Text string1;
SimpleStruct(String b1, String s1) {
this.k = new Text(b1);
if (s1 == null) {
this.string1 = null;
} else {
this.string1 = new Text(s1);
}
}
}
然后在测试方法中,我阅读了它并使用mrunitinvokemapper。下面是代码
// Read orc file
Reader reader = OrcFile.createReader(fs, testFilePath) ;
RecordReader recordRdr = reader.rows() ;
OrcStruct row = null ;
List<OrcStruct> mapData = new ArrayList<>()
while(recordRdr.hasNext()) {
row = (OrcStruct) recordRdr.next(row) ;
mapData.add(row) ;
}
// test mapper
initializeSerde(mapDriver.getConfiguration());
Writable writable = getWritable(mapData.get(0)) ; // test 1st record's mapper processing
mapDriver.withCacheFile(strCachePath).withInput(NullWritable.get(), writable );
mapDriver.runTest();
查看orcserde,我可以看到MRUnit调用的write不受支持。因此,测试用例错误被排除
我们如何对正在处理Orc文件的映射器进行单元测试。在我所做的事情中是否有其他方式或需要改变什么
提前谢谢你的帮助
java.lang.UnsupportedOperationException: can't write the bundle
at org.apache.hadoop.hive.ql.io.orc.OrcSerde$OrcSerdeRow.write(OrcSerde.java:61)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:98)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:82)
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:80)
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:110)
at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:675)
at org.apache.hadoop.mrunit.TestDriver.copyPair(TestDriver.java:679)
at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:120)
at org.apache.hadoop.mrunit.MapDriverBase.withInput(MapDriverBase.java:210)