Hadoop 通过在清管器中创建大量过滤器

我有这个密码

large = load 'a super large file' 

CC = FILTER large BY $19 == 'abc OR $20 == 'abc' 
OR $19 == 'def' or $20 == 'def' ....;

条件的数量可以达到100甚至数千

---

有更好的方法吗？

是的，将这些条件放在另一个文件中。将其加载到一个关系中，并在列上连接两个关系。如果必须在多个列上进行筛选，则创建与条件一样多的筛选文件。下面是两列的示例

large = load 'a super large file' 
filter1 = load 'file with values needed to compare with $19';
filter2 = load 'file with values needed to compare with $20';
f1 = JOIN large BY $19,filter1 BY $0;
f2 = JOIN large BY $20,filter2 BY $0;
final = UNION f1,f2;
DUMP final;

您可能可以使用一个包含多个列的筛选文件，并对这些列进行联接，以获得不同的筛选结果，然后只需合并关系即可

large = load 'a super large file' 
filter_file = load 'file with values in different columns';

f1 = JOIN large BY $19,filter_file BY $0;
f2 = JOIN large BY $20,filter_file BY $1;
final = UNION f1,f2;
DUMP final;

---

是的，将这些条件放在另一个文件中。将其加载到一个关系中，并将列上的两个关系连接起来。如果必须对多个列进行筛选，则创建与条件数量相同的筛选文件。下面是两列的示例

large = load 'a super large file' 
filter1 = load 'file with values needed to compare with $19';
filter2 = load 'file with values needed to compare with $20';
f1 = JOIN large BY $19,filter1 BY $0;
f2 = JOIN large BY $20,filter2 BY $0;
final = UNION f1,f2;
DUMP final;

您可能可以使用一个包含多个列的筛选文件，并对这些列进行联接，以获得不同的筛选结果，然后只需合并关系即可

large = load 'a super large file' 
filter_file = load 'file with values in different columns';

f1 = JOIN large BY $19,filter_file BY $0;
f2 = JOIN large BY $20,filter_file BY $1;
final = UNION f1,f2;
DUMP final;