Spark Scala-过滤器时间戳

我正在尝试使用Spark Scala忽略日期来过滤两个值之间的时间戳。我试图只选择9:00:00pm和11:00:00pm之间的所有记录，包括9:00:00和11:00:00。下面列出了我当前的输入、输出和代码

我的思想过程是能够使用大于或小于我的值的pickupWindow列进行过滤

有什么想法吗

输入：

+----------------------+----------------------+----------+------------+------------+
|tpep_pickup_datetime  |tpep_dropoff_datetime |total_amount|pickupWindow|
+----------------------+----------------------+----------+------------+------------+
|05/18/2018 09:09:29 PM|05/18/2018 09:52:53 PM|42.8        |09:09:29    |
|05/18/2018 11:00:00 PM|05/18/2018 11:09:13 PM|23.5        |11:00:00    |
|05/18/2018 02:47:21 PM|05/18/2018 03:30:00 PM|46.62       |02:47:21    |

电流输出：

+--------------------+---------------------+---------+------------+------------+
|tpep_pickup_datetime|tpep_dropoff_datetime|timestamp|total_amount|pickupWindow|
+--------------------+---------------------+---------+------------+------------+
+--------------------+---------------------+---------+------------+------------+

当前代码：

stamp.withColumn("pickupWindow",date_format(to_timestamp(col("tpep_pickup_datetime"),"MM/dd/yyyy hh:mm:ss a"),"hh:mm:ss")).select("tpep_pickup_datetime","tpep_dropoff_datetime","timestamp","total_amount","pickupWindow").filter(col("pickupWindow")>="9:00:00").filter(col("pickupWindow")<="11:00:00").where($"tpep_pickup_datetime".contains("PM")).show(false)

尝试使用.geq和.leq，如下所示-

scala> df.withColumn("pickupWindow",date_format(to_timestamp(col("tpep_pickup_datetime"),"MM/dd/yyyy hh:mm:ss a"),"hh:mm:ss")).filter(col("pickupWindow").geq("09:00:00") && col("pickupWindow").leq("11:00:00")).
    show()

    +----------------------+----------------------+---------+------------+------------+
    |tpep_pickup_datetime  |tpep_dropoff_datetime |timestamp|total_amount|pickupWindow|
    +----------------------+----------------------+---------+------------+------------+
    |05/18/2018 09:56:20 PM|05/18/2018 10:50:38 PM|35780    |52.87       |09:56:20    |
    |05/18/2018 10:52:49 PM|05/18/2018 11:08:47 PM|39169    |14.76       |10:52:49    |
    |05/18/2018 09:01:22 PM|05/18/2018 09:05:36 PM|32482    |6.3         |09:01:22    |
    |05/18/2018 09:00:29 PM|05/18/2018 09:05:31 PM|32429    |7.56        |09:00:29    |
    +----------------------+----------------------+---------+------------+------------+

如果你想要AM&PM转换

scala> df.withColumn("pickupWindow",date_format(to_timestamp(col("tpep_pickup_datetime"),"MM/dd/yyyy hh:mm:ss a"),"HH:mm:ss")).filter(col("pickupWindow").geq("21:00:00") && col("pickupWindow").leq("23:00:00")).
    show(false)
    +----------------------+----------------------+---------+------------+------------+
    |tpep_pickup_datetime  |tpep_dropoff_datetime |timestamp|total_amount|pickupWindow|
    +----------------------+----------------------+---------+------------+------------+
    |05/18/2018 09:56:20 PM|05/18/2018 10:50:38 PM|35780    |52.87       |21:56:20    |
    |05/18/2018 10:52:49 PM|05/18/2018 11:08:47 PM|39169    |14.76       |22:52:49    |
    |05/18/2018 09:01:22 PM|05/18/2018 09:05:36 PM|32482    |6.3         |21:01:22    |
    |05/18/2018 09:00:29 PM|05/18/2018 09:05:31 PM|32429    |7.56        |21:00:29    |
    +----------------------+----------------------+---------+------------+------------+