Apache pig 访问日志检查作业。计数和不同计数

我有HTTP访问日志的TBs，我正试图分析这些日志以供历史参考。我正在尝试获取每日唯一访客（IP）的数量和每日点击次数

如果这是在MySQL中，我会这样做：

SELECT COUNT(1) from access_logs union
SELECT COUNT(DISTINCT(ip)) from access_logs

X = load 'path' as (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
IPs = FOREACH X GENERATE ip; // project early for performance reasons
Dist_IPs = distinct IPs;
GRP_DIST = group Dist_IPs all;
DIST = foreach GRP_DIST generate COUNT(GRP_DIST) as cnt, 'dist' as category;

GRP_ALL = group IPs all;
ALL = foreach GRP_ALL generate COUNT(GRP_ALL)as cnt, 'all' as category;

OUT = union DIST, ALL;

有没有办法用一份工作做类似的事情，这样我就不必再做第二次地图工作了

我知道这不起作用，但这是我正在寻找的功能

X = (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
Y = COUNT(X);
Z = COUNT(DISTINCT(X.IP);
OUT = UNION Y,Z;
STORE OUT into ...

---

如果要计算输入中的所有记录，则需要使用GROUP all，这将创建一个包。出于性能原因，它还使用了累加器独特的函数org.apache.pig.builtin.DISTINCT

X = load 'path' as (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
IPs = FOREACH X GENERATE ip; // project early for performance reasons
GRP = group IPs all;
OUT = foreach GRP generate COUNT(IPs) as all_cnt, COUNT(org.apache.pig.builtin.Distinct(IPs.ip)) as distinct_cnt;

如果IP太多，并且出现与内存相关的异常，则可以执行以下操作：

SELECT COUNT(1) from access_logs union
SELECT COUNT(DISTINCT(ip)) from access_logs

X = load 'path' as (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
IPs = FOREACH X GENERATE ip; // project early for performance reasons
Dist_IPs = distinct IPs;
GRP_DIST = group Dist_IPs all;
DIST = foreach GRP_DIST generate COUNT(GRP_DIST) as cnt, 'dist' as category;

GRP_ALL = group IPs all;
ALL = foreach GRP_ALL generate COUNT(GRP_ALL)as cnt, 'all' as category;

OUT = union DIST, ALL;