Hadoop 如何使用ApachePig将标题行连接到多个文件中的详细行

Hadoop 如何使用ApachePig将标题行连接到多个文件中的详细行,hadoop,apache-pig,Hadoop,Apache Pig,我在一个HDFS文件夹中有几个CSV文件,我将其加载到一个关系中: source=使用PigStorage(',')加载“$data”--$data作为参数传递给pig命令 当我转储它时,源关系的结构如下:(注意数据是文本限定的,但我将使用REPLACE函数处理) 其中每个头元组后面跟着一包属于该头的记录元组?。 不幸的是,标题和细节行之间没有公共键字段,所以我认为不能使用任何连接操作 我对Pig和Hadoop很陌生,这是我参与的第一个概念项目之一 希望我的问题很清楚,并期待在这里得到一些指导。

我在一个HDFS文件夹中有几个CSV文件,我将其加载到一个关系中:

source=使用PigStorage(',')加载“$data”--$data作为参数传递给pig命令

当我转储它时,源关系的结构如下:(注意数据是文本限定的,但我将使用REPLACE函数处理)

其中每个头元组后面跟着一包属于该头的记录元组?。 不幸的是,标题和细节行之间没有公共键字段,所以我认为不能使用任何连接操作

我对Pig和Hadoop很陌生,这是我参与的第一个概念项目之一


希望我的问题很清楚,并期待在这里得到一些指导。

这应该会让你开始。
代码:


是的,这确实让我开始了,谢谢你!!
("HEADER","20110118","20101218","20110118","T00002")
("0000000000000000035412","20110107","2699","D","20110107","2315.","","","","","","C")
("0000000000000000035412","20110107","2699","D","20110107","246..","162","74","","","","B")

<.... more records ....>

("HEADER","20110224","20110109","20110224","T00002")
("0000000000000000035412","20110121","2028","D","20110121","a6c3.","","","","","R","P")
("0000000000000000035412","20110217","2619","D","20110217","a6c3.","","","","","R","P")

<.... more records ....>
{
(HEADER,20110118,20101218,20110118,T00002),{(0000000000000000035412,20110107,2699,D,20110107,2315.,,,,,,C),(0000000000000000035412,20110107,2699,D,20110107,246..,162,74,,,,B),..more tuples..},
(HEADER,20110224,20110109,20110224,T00002),{(0000000000000000035412,20110121,2028,D,20110121,a6c3.,,,,,R,P),(0000000000000000035412,20110217,2619,D,20110217,a6c3.,,,,,R,P),..more tuples..},..more tuples..
}
Source = LOAD '$data' USING PigStorage(',','-tagFile');
A = SPLIT Source INTO FileHeaders IF $1 == 'HEADER', FileData OTHERWISE;
B = GROUP FileData BY $0;
C = GROUP FileHeaders BY $0;
D = JOIN B BY Group, C BY Group;
...