Hadoop 使用自定义换行符加载CSV

我有一个CSV，如下所示，行以“+++”而不是新行终止。如何通过在字符串“+++”存在的地方换行来加载csv

VTS,51,0071,9739965515,NM,GP,INF01,V,19,072219,291014,0000.0000,N,00000.0000,E,07AE

VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205+++VTS,51,0071,9739965515,NM,GP,INF01,V,18,072311,291014,0000.0000,N,00000.0000,E,C24E+++VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358

VTS,51,0071,9739965515,NM,GP,INF01,V,18,072319,291014,0000.0000,N,00000.0000,E,012F
VTS,51,0071,9739965515,NM,GP,INF01,V,19,072326,291014,0000.0000,N,00000.0000,E,B2E6+++VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0
VTS,51,0071,9739965515,NM,GP,INF01,V,18,072333,291014,0000.0000,N,00000.0000,E,9896
VTS,51,0071,9739965515,NM,GP,INF01,V,18,072340,291014,0000.0000,N,00000.0000,E,9B23

首先，我需要在出现新行或“+++”符号的地方断开该行，然后加载数据。然后，再次使用第二列中的值01进行过滤

预期输出：

VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205
VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358
VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0

A = LOAD 'input.csv' AS (line:chararray);
B = FOREACH A {
                 splitRow = TOKENIZE(line,'+++');
                 GENERATE FLATTEN(splitRow) AS newList;
              }
C = FOREACH B GENERATE FLATTEN(STRSPLIT(newList,',',16));
D = FILTER C BY $1==01;
DUMP D;

(VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205)
(VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358)
(VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0)

---

PigScript:

VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205
VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358
VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0

A = LOAD 'input.csv' AS (line:chararray);
B = FOREACH A {
                 splitRow = TOKENIZE(line,'+++');
                 GENERATE FLATTEN(splitRow) AS newList;
              }
C = FOREACH B GENERATE FLATTEN(STRSPLIT(newList,',',16));
D = FILTER C BY $1==01;
DUMP D;

(VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205)
(VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358)
(VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0)

输出：

VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205
VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358
VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0

A = LOAD 'input.csv' AS (line:chararray);
B = FOREACH A {
                 splitRow = TOKENIZE(line,'+++');
                 GENERATE FLATTEN(splitRow) AS newList;
              }
C = FOREACH B GENERATE FLATTEN(STRSPLIT(newList,',',16));
D = FILTER C BY $1==01;
DUMP D;

(VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205)
(VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358)
(VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0)

---

您的预期输出是什么？@SivasakthiJayaraman预期输出已更新解决方案，请验证并让我知道这是否适用于您。请解释上述步骤，以便我能够理解当我添加额外的行E=foreach D生成$7，$15；倾倒区E；我得到的结果是（060037061114068,004000,0099914994）（060113061114068,004000,00999152，B927）为什么不必要地将所有这些文件的其余部分都改为$7和$15，并更改行“C=FOREACH B GENERATE flatte（STRSPLIT（newList，，，，，，，，，，，”；“即，代替16，给出23。基本上是它的总列数。我错给了16分。