Dataframe PySpark Filter between-提供基于组的上限和下限列表

我有一个PySpark数据帧，并希望筛选介于上限和下限之间的行。 通常，我只会使用一个介于以下两者之间的过滤器：

import pandas as pd
from pyspark.sql import functions as F
... sql_context creation ...

pdfRaw=pd.DataFrame([{"vehicleID":'A', "Segment":'State Hwy', "speed":68.0},\
{"vehicleID":'B', "Segment":'State Hwy', "speed":76.0}])
dfRaw = sql_context.createDataFrame(pdfRaw).withColumn("vehicleID", "Segment", "speed")

dfRaw.show()

+-----------+------------+-----+
   vehicleID|     Segment|value|
+-----------+------------+-----+
|          A|   State Hwy| 68.0|
|          B|   State Hwy| 73.0|
+-----------+------------+-----+

dfRaw.filter(F.col("speed").between(70,75)).show()

+-----------+------------+-----+
   vehicleID|     Segment|value|
+-----------+------------+-----+
|          B|   State Hwy| 73.0|
+-----------+------------+-----+

但是，我有多个速度值，我想在它们之间进行过滤

Speeds_Curious = {
[25,30],
[55,60],
[60,65],
[70,75]
}

实际上我想更进一步。过滤器的上下限取决于前一个数据帧的groupby结果

df_RoadSegments.groupby('Segment')\
.agg(F.min('SpeedLimit').alias('minSpeed'),\
F.max('SpeedLimit').alias('maxSpeed'))\
.show()

+-----------+----------+----------+
     Segment|  minSpeed|  maxSpeed|
+-----------+----------+----------+
|      Urban|      25.0|      30.0|
|  State Hwy|      55.0|      60.0|
|I-State Hwy|      60.0|      65.0|
|I-State Hwy|      70.0|      75.0|
+-----------+----------+----------+

所以基本上我想在不同数据帧上可用作列的值之间过滤数据帧

比如：

dfLimits = df_RoadSegments.groupby('Segment')\
.agg(F.min('SpeedLimit').alias('minSpeed'),\ F.max('SpeedLimit').alias('maxSpeed'))

dfRaw.groupby('Segment')\
.filter(F.col("speed")\
.between(dfLimits.where(dfLimits.Segment=="State Hwy"(??)).select('minSpeed')),\
dfLimits.where(dfLimits.Segment=="State Hwy"(??)).select('maxSpeed'))))\
.show()

---

有什么想法吗？

以下方法将为您提供所属特定路段的

min

和

max

速度之间的所有车辆

您可以连接两个数据帧：

df_joined = dfRaw.join(dfLimits, on="Segment", how="left")
+---------+---------+-----+--------+--------+
|  Segment|vehicleID|speed|minSpeed|maxSpeed|
+---------+---------+-----+--------+--------+
|State Hwy|        A| 68.0|      55|      60|
|State Hwy|        B| 76.0|      55|      60|
+---------+---------+-----+--------+--------+

如果您想进一步标记速度是否在上述界限之间，则可以写入：

flag_df = df_joined.withColumn("flag", F.when((F.col("speed") > F.col("minSpeed")) & (F.col("speed") < F.col("minSpeed")), 1).otherwise(0))
flag_df.show()
+---------+---------+-----+--------+--------+----+
|  Segment|vehicleID|speed|minSpeed|maxSpeed|flag|
+---------+---------+-----+--------+--------+----+
|State Hwy|        A| 68.0|      55|      60|   0|
|State Hwy|        B| 76.0|      55|      60|   0|
+---------+---------+-----+--------+--------+----+

---

好的，这适用于原始问题。但还有一个警告。对于给定的“段”，可能有多个“分钟速度”和“最大速度”。我编辑了这个问题，提到了“I-statehway”段的两个条件。@penguin这将按行执行，只需更改

join=“outer”

，以便它复制段soh的行！是的，应该这样。现在将你的回答标记为答案。

df_final = df.filter(F.col("flag") == 1)