Apache pig Pig UDF-将动态模式作为一组字段(而不是元组)返回
在GROUP BY+PLANTE之后,我有一个名称空间为的数据:Apache pig Pig UDF-将动态模式作为一组字段(而不是元组)返回,apache-pig,jython,user-defined-functions,Apache Pig,Jython,User Defined Functions,在GROUP BY+PLANTE之后,我有一个名称空间为的数据: DESCRIBE users; users: {user_id: int, group_id: int, registration_timestamp: int} users_with_namespace = FOREACH (GROUP users BY group_id) { first_to_latest = ORDER users BY registration_timestamp ASC; first_
DESCRIBE users;
users: {user_id: int, group_id: int, registration_timestamp: int}
users_with_namespace = FOREACH (GROUP users BY group_id) {
first_to_latest = ORDER users BY registration_timestamp ASC;
first_user = LIMIT first_to_latest 1;
GENERATE FLATTEN(first_user);
};
DESCRIBE users_with_namespace;
users_with_namespace: {first_user::user_id: int, first_user::group_id: int, first_user::registration_timestamp: int}
我希望能够做到以下几点:
users = myudf.strip_namespace(users_with_namespace);
或者(因为,这似乎不可能):
结果是:
> DESCRIBE users;
users: {user_id: int, registration_timestamp: int}
我已经编写了一个Jython Pig UDF,它应该去掉任何名称空间的字段名,但我似乎无法从我的UDF返回一组字段。只有一个包/元组/单个字段是可能的,这给我留下了如下结果:
DESCRIBE users;
users: {t: (user_id: int, registration_timestamp: int)}
@outputSchemaFunction("tupleSchema")
def strip_namespace(input):
return input
@schemaFunction("tupleSchema")
def tupleSchema(input):
fields = []
dt = []
for i in input.getField(0).schema.getFields():
for field in i.schema.getFields():
fields.append(field.alias.split("::")[-1])
dt.append(field.type)
return SchemaUtil.newTupleSchema(fields, dt)
有没有办法省略“t”并返回字段列表/集合?我的UDF如下所示:
DESCRIBE users;
users: {t: (user_id: int, registration_timestamp: int)}
@outputSchemaFunction("tupleSchema")
def strip_namespace(input):
return input
@schemaFunction("tupleSchema")
def tupleSchema(input):
fields = []
dt = []
for i in input.getField(0).schema.getFields():
for field in i.schema.getFields():
fields.append(field.alias.split("::")[-1])
dt.append(field.type)
return SchemaUtil.newTupleSchema(fields, dt)
到目前为止,我已经使用了
FOREACH .. GENERATE namespace::field as field, ...
剥离名称空间,但这种方法对于具有许多字段的数据集来说确实很乏味。不幸的是,您不能,至少现在不能。问题正是您所说的:现在您只能返回一个元组、一个包或单个字段。我在2个月前创建了一个允许返回此场景的多个字段,但还没有回复 我真的希望他们在将来实现这一点,因为当您必须执行许多连接时,您最终会得到比实际代码更多的
FOREACH
语句来重命名字段