Python 3.x 如果csv文件在Pyspark中包含特定关键字，则跳过该文件中的行

我有一个CSV文件，详细信息如下所示：

emp_id,emp_name,emp_city,emp_salary
1,VIKRANT SINGH RANA    ,NOIDA   ,10000
3,GOVIND NIMBHAL        ,DWARKA  ,92000
2,RAGHVENDRA KUMAR GUPTA,GURGAON ,50000
4,ABHIJAN SINHA         ,SAKET   ,65000
5,SUPER DEVELOPER       ,USA     ,50000
6,RAJAT TYAGI           ,UP      ,65000
7,AJAY SHARMA           ,NOIDA   ,70000
8,SIDDHARTH BASU        ,SAKET   ,72000
9,ROBERT                ,GURGAON ,70000
9,ABC                   ,ROBERT  ,10000
9,XYZ                   ,ROBERTGURGAON,70000

如果包含关键字“ROBERT”且预期输出为：

+------+--------------------+-------------+----------+
|emp_id|            emp_name|     emp_city|emp_salary|
+------+--------------------+-------------+----------+
|     1|VIKRANT SINGH RAN...|     NOIDA   |     10000|
|     3|GOVIND NIMBHAL   ...|     DWARKA  |     92000|
|     2|RAGHVENDRA KUMAR ...|     GURGAON |     50000|
|     4|ABHIJAN SINHA    ...|     SAKET   |     65000|
|     5|SUPER DEVELOPER  ...|     USA     |     50000|
|     6|RAJAT TYAGI      ...|     UP      |     65000|
|     7|AJAY SHARMA      ...|     NOIDA   |     70000|
|     8|SIDDHARTH BASU   ...|     SAKET   |     72000|
+------+--------------------+-------------+----------+

我可以将这个文件加载到dataframe中，并可以对每一列使用下面的表达式进行过滤

newdf = emp_df.where(~ col("emp_city").like("ROBERT%"))

---

我正在寻找一些解决方案，以便在将其加载到dataframe之前对其进行过滤，而无需遍历所有列来查找特定字符串。

我能够使用RDD对其进行过滤

textdata = sc.textFile(PATH_TO_FILE)
header=textdata.first();
textnewdata = textdata.filter(lambda x:x != header)
newRDD = textnewdata.filter(lambda row : 'ROBERT' not in row)

[u'1,VIKRANT SINGH RANA    ,NOIDA   ,10000', 
u'3,GOVIND NIMBHAL        ,DWARKA  ,92000', 
u'2,RAGHVENDRA KUMAR GUPTA,GURGAON ,50000', 
u'4,ABHIJAN SINHA         ,SAKET   ,65000', 
u'5,SUPER DEVELOPER       ,USA     ,50000', 
u'6,RAJAT TYAGI           ,UP      ,65000', 
u'7,AJAY SHARMA           ,NOIDA   ,70000', 
u'8,SIDDHARTH BASU        ,SAKET   ,72000']

newsplitRDD = newRDD.map(lambda l: l.split(","))

newDF = newsplitRDD.toDF()

>>> newDF.show();
+---+--------------------+--------+-----+
| _1|                  _2|      _3|   _4|
+---+--------------------+--------+-----+
|  1|VIKRANT SINGH RAN...|NOIDA   |10000|
|  3|GOVIND NIMBHAL   ...|DWARKA  |92000|
|  2|RAGHVENDRA KUMAR ...|GURGAON |50000|
|  4|ABHIJAN SINHA    ...|SAKET   |65000|
|  5|SUPER DEVELOPER  ...|USA     |50000|
|  6|RAJAT TYAGI      ...|UP      |65000|
|  7|AJAY SHARMA      ...|NOIDA   |70000|
|  8|SIDDHARTH BASU   ...|SAKET   |72000|
+---+--------------------+--------+-----+

---

我能够使用RDD对其进行过滤

textdata = sc.textFile(PATH_TO_FILE)
header=textdata.first();
textnewdata = textdata.filter(lambda x:x != header)
newRDD = textnewdata.filter(lambda row : 'ROBERT' not in row)

[u'1,VIKRANT SINGH RANA    ,NOIDA   ,10000', 
u'3,GOVIND NIMBHAL        ,DWARKA  ,92000', 
u'2,RAGHVENDRA KUMAR GUPTA,GURGAON ,50000', 
u'4,ABHIJAN SINHA         ,SAKET   ,65000', 
u'5,SUPER DEVELOPER       ,USA     ,50000', 
u'6,RAJAT TYAGI           ,UP      ,65000', 
u'7,AJAY SHARMA           ,NOIDA   ,70000', 
u'8,SIDDHARTH BASU        ,SAKET   ,72000']

newsplitRDD = newRDD.map(lambda l: l.split(","))

newDF = newsplitRDD.toDF()

>>> newDF.show();
+---+--------------------+--------+-----+
| _1|                  _2|      _3|   _4|
+---+--------------------+--------+-----+
|  1|VIKRANT SINGH RAN...|NOIDA   |10000|
|  3|GOVIND NIMBHAL   ...|DWARKA  |92000|
|  2|RAGHVENDRA KUMAR ...|GURGAON |50000|
|  4|ABHIJAN SINHA    ...|SAKET   |65000|
|  5|SUPER DEVELOPER  ...|USA     |50000|
|  6|RAJAT TYAGI      ...|UP      |65000|
|  7|AJAY SHARMA      ...|NOIDA   |70000|
|  8|SIDDHARTH BASU   ...|SAKET   |72000|
+---+--------------------+--------+-----+

---

在创建数据帧之前，您是否尝试使用

lambda

函数将其读入

rdd

和

filter

结果？类似这样：

rdd.filter（lambda行：'ROBERT'不在行中）。toDF（rdd.first（））

是的，它可以工作。感谢Shappy听到它的帮助：）并感谢您为其编写了正确的答案。在创建数据帧之前，您是否尝试使用

lambda

函数将其读入

rdd

并

filter

结果？类似这样：

rdd.filter（lambda行：'ROBERT'不在行中）。toDF（rdd.first（））

是的，它可以工作。谢谢你听到它的帮助：）谢谢你为它写了一个正确的答案。