Hadoop Apache Pig无法获取MIN(列),意外符号
在这种情况下,我正在努力弄清楚如何使用MIN()函数。我有以下的猪脚本:Hadoop Apache Pig无法获取MIN(列),意外符号,hadoop,apache-pig,min,Hadoop,Apache Pig,Min,在这种情况下,我正在努力弄清楚如何使用MIN()函数。我有以下的猪脚本: A = LOAD '/home/mqp/Documents/p1/data/test_customers.csv' USING CSVExcelStorage (',') AS (custid:int, name:chararray, age:int, gender:chararray, country:int, salary:float); B = LOAD '/home/mqp/Documents/p1/data/
A = LOAD '/home/mqp/Documents/p1/data/test_customers.csv' USING CSVExcelStorage (',') AS
(custid:int, name:chararray, age:int, gender:chararray, country:int, salary:float);
B = LOAD '/home/mqp/Documents/p1/data/test_transactions.csv' USING CSVExcelStorage (',') AS (transid:int, custid:int, ttotal:float, items:int, tdesc:chararray);
C = JOIN B BY custid, A BY custid USING 'replicated';
D = GROUP C BY $1;
DESCRIBE D;
out = FOREACH D {
allids = FOREACH C GENERATE B::custid;
singleids = DISTINCT allids;
allnames = FOREACH C GENERATE name;
singlenames = DISTINCT allnames;
allsal= FOREACH C GENERATE salary;
singlesal = DISTINCT allsal;
alltotals = FOREACH C GENERATE B::ttotal as bt;
mintotals = FOREACH alltotals GENERATE MIN(alltotals.bt);
transtotal = FOREACH C GENERATE ttotal;
GENERATE flatten(singleids), flatten(singlenames), flatten(singlesal), COUNT(C), SUM(transtotal), flatten(mintotals);
};
STORE out INTO '/home/mqp/Documents/p1/pig_test' USING CSVExcelStorage();
我尝试了无数种不同的方法来让MIN()函数在这里工作,但都无济于事。我尝试过按所有人分组,使用不同的索引,等等。我真的不明白我需要做什么
我收到错误“foo处或附近出现意外符号”和“无效标量投影”您应该在主
GENERATE
块中使用MIN
函数,而不是在嵌套的FOREACH
中使用它。这是因为MIN函数需要一包值,如正确使用的COUNT
和SUM
下面是一个可能的脚本版本,应该可以使用(为了清晰起见,我对您的原始脚本进行了一些更改,并添加了一些注释):
此脚本是否在不使用MIN函数的情况下工作?
A = LOAD 'test_customers.csv' USING CSVExcelStorage (',') AS (
custid:int,
name:chararray,
age:int,
gender:chararray,
country:int,
salary:float
);
B = LOAD 'test_transactions.csv' USING CSVExcelStorage (',') AS (
transid:int,
custid:int,
ttotal:float,
items:int,
tdesc:chararray
);
C = JOIN B BY custid, A BY custid USING 'replicated';
-- Removed the DESCRIBE and used field name in GROUP BY for clarity.
D = GROUP C BY B::custid;
out = FOREACH D {
-- Removed DISTINCT custid
-- Because we grouped by this field, we can just generate group
-- Consider adding name and salary to the GROUP BY if you
-- expect these to be the same for each custid.
allnames = FOREACH C GENERATE name;
singlenames = DISTINCT allnames;
allsal= FOREACH C GENERATE salary;
singlesal = DISTINCT allsal;
alltotals = FOREACH C GENERATE B::ttotal as bt;
-- transtotals was effectively the same as alltotals
GENERATE
group,
FLATTEN(singlenames),
FLATTEN(singlesal),
COUNT(C),
SUM(alltotals.bt),
MIN(alltotals.bt);
}
STORE out INTO 'pig_test' USING CSVExcelStorage();