Java 拼花文件可选字段不存在
我是拼花文件的新手,我想开发一个mapreduce作业,用以下shcema读取许多输入拼花文件:Java 拼花文件可选字段不存在,java,hadoop,mapreduce,parquet,Java,Hadoop,Mapreduce,Parquet,我是拼花文件的新手,我想开发一个mapreduce作业,用以下shcema读取许多输入拼花文件: { optional int96 dropoff_datetime; optional float dropoff_latitude; optional float dropoff_longitude; optional int32 dropoff_taxizone_id; optional float ehail_fee; optional float extra; o
{
optional int96 dropoff_datetime;
optional float dropoff_latitude;
optional float dropoff_longitude;
optional int32 dropoff_taxizone_id;
optional float ehail_fee;
optional float extra;
optional float fare_amount;
optional float improvement_surcharge;
optional float mta_tax;
optional int32 passenger_count;
optional binary payment_type (UTF8);
optional int96 pickup_datetime;
optional float pickup_latitude;
optional float pickup_longitude;
optional int32 pickup_taxizone_id;
optional int32 rate_code_id;
optional binary store_and_fwd_flag (UTF8);
optional float tip_amount;
optional float tolls_amount;
optional float total_amount;
optional float trip_distance;
optional binary trip_type (UTF8);
optional binary vendor_id (UTF8);
required int64 trip_id;
}
我的工作目标是计算每天每小时的平均出行速度,因此我需要提取所有出行距离和上下车时间,以计算持续时间和速度,但是当字段出行距离
不存在时,我会出错:这里是堆栈跟踪的一部分:
18/02/28 03:19:01 INFO mapreduce.Job: map 2% reduce 0%
18/02/28 03:19:10 INFO mapreduce.Job: Task Id : attempt_1519722054260_0016_m_000011_2, Status : FAILED
Error: java.lang.RuntimeException: not found 20(trip_distance) element number 0 in group:
dropoff_datetime: Int96Value{Binary{12 constant bytes, [0, 0, 0, 0, 0, 0, 0, 0, -116, 61, 37, 0]}}
payment_type: ""
pickup_datetime: Int96Value{Binary{12 constant bytes, [0, 120, 66, 9, 78, 72, 0, 0, 3, 125, 37, 0]}}
pickup_latitude: 40.7565
pickup_longitude: -73.9781
pickup_taxizone_id: 161
store_and_fwd_flag: ""
trip_type: "uber"
vendor_id: ""
trip_id: 4776003633207
at org.apache.parquet.example.data.simple.SimpleGroup.getValue(SimpleGroup.java:97)
at org.apache.parquet.example.data.simple.SimpleGroup.getValueToString(SimpleGroup.java:119)
at ParquetAssignmentSpeedAverageHours$ParquetMap.map(ParquetAssignmentSpeedAverageHours.java:48)
at ParquetAssignmentSpeedAverageHours$ParquetMap.map(ParquetAssignmentSpeedAverageHours.java:37)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
这是我的mapper类:
public static class ParquetMap extends Mapper<Text, Group, IntWritable, DoubleWritable> {
private DoubleWritable one = new DoubleWritable(1);
private IntWritable time = new IntWritable();
private DoubleWritable result = new DoubleWritable();
@Override
public void map(Text key, Group value, Context context) throws IOException, InterruptedException {
double duration;
double distance;
double speed;
Binary pickupTimestamp = value.getInt96("pickup_datetime", 0);
Binary dropoffTimestamp = value.getInt96("dropoff_datetime", 0);
if (value.getValueToString(20, 0) != null) { //the trip_distance field
distance = value.getFloat("trip_distance", 0);
} else {
distance = 0;
}
try {
if (!pickupTimestamp.equals(dropoffTimestamp)) {
duration = ((double)(getTimestampMillis(dropoffTimestamp) - getTimestampMillis(pickupTimestamp))/3600000);
speed = (float) (distance / duration);
result.set((speed));
Calendar cal = Calendar.getInstance();
cal.setTimeInMillis(getTimestampMillis(pickupTimestamp));
time.set(cal.get(Calendar.HOUR_OF_DAY));
context.write(time, result);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
公共静态类ParquetMap扩展映射器{
私有双可写1=新的双可写(1);
private intwriteable time=new intwriteable();
私有DoubleWritable结果=新的DoubleWritable();
@凌驾
公共void映射(文本键、组值、上下文上下文)引发IOException、InterruptedException{
双倍持续时间;
双倍距离;
双速;
二进制pickupTimestamp=value.getInt96(“picku_datetime”,0);
二进制dropoffTimestamp=value.getInt96(“dropoff_datetime”,0);
如果(value.getValueToString(20,0)!=null){//行程距离字段
距离=value.getFloat(“跳闸距离”,0);
}否则{
距离=0;
}
试一试{
如果(!pickupTimestamp.equals(dropoffTimestamp)){
持续时间=((双精度)(getTimestampMillis(dropoffTimestamp)-getTimestampMillis(pickupTimestamp))/3600000;
速度=(浮动)(距离/持续时间);
结果。设置((速度));
Calendar cal=Calendar.getInstance();
cal.setTimeInMillis(getTimestampMillis(pickupTimestamp));
time.set(cal.get(日历小时/天));
写(时间、结果);
}
}捕获(例外e){
e、 printStackTrace();
}
}
}
有人能帮忙吗?
谢谢,这是一个运行时异常
java.lang.RuntimeException
;这基本上表明代码有缺陷
公共字符串getValueToString(int-fieldIndex,int-index)
方法在内部调用getValue(int-fieldIndex,int-index)
方法。
getValue(…)
的实现如下
private Object getValue(int fieldIndex, int index) {
List<Object> list;
try {
list = data[fieldIndex];
} catch (IndexOutOfBoundsException e) {
throw new RuntimeException("not found " + fieldIndex + "(" + schema.getFieldName(fieldIndex) + ") in group:\n" + this);
}
try {
return list.get(index);
} catch (IndexOutOfBoundsException e) {
throw new RuntimeException("not found " + fieldIndex + "(" + schema.getFieldName(fieldIndex) + ") element number " + index + " in group:\n" + this);
}
}
添加了可能的原因以及您可以如何处理它。希望能有所帮助。如果答案已经解决了你的问题,请考虑一下投票。
try{
distance = value.getFloat("trip_distance", 0);
}catch(RuntimeException e){
distance = 0;
}