Python Pyspark-操纵数据帧以获取状态的时间变化

我在Pyspark中有一个数据帧，描述服务的状态，如下所示。我接收服务状态更新的频率不是恒定的

地位 更新的时间戳 好啊 2020-01-01 14:30:00 好啊 2020-01-01 14:15:00 破碎的 2020-01-01 14:10:00 破碎的 2020-01-01 14:00:00 破碎的 2020-01-01 13:40:00 破碎的 2020-01-01 13:35:00 好啊 2020-01-01 13:15:00 好啊 2020-01-01 13:00:00 好啊 2020-01-01 12:40:00

---

您可以创建一些帮助器列来检查所需的条件：

from pyspark.sql import functions as F, Window

df2 = df.withColumn(
    'first_row',
    F.row_number().over(Window.orderBy('timestamp_of_update')) == 1
).withColumn(
    'change_to_ok', 
    (F.lag('status').over(Window.orderBy('timestamp_of_update')) != 'OK') & 
    (F.col('status') == 'OK') 
).withColumn(
    'last_ok', 
    F.last(
        F.when(F.col('status') == 'OK', F.col('timestamp_of_update')), 
        True
    ).over(Window.orderBy('timestamp_of_update'))
).withColumn(
    'time',
    F.when(
        F.col('status') == 'Broken',
        F.col('last_ok')
    ).when(
        F.col('change_to_ok'),
        F.col('timestamp_of_update')
    ).when(
        F.col('status') == 'OK',
        F.lag('timestamp_of_update').over(Window.orderBy('timestamp_of_update'))
    )
).withColumn(
    'time_gone_by', 
    (F.unix_timestamp('timestamp_of_update') - F.unix_timestamp('time'))/60
).select('status', 'timestamp_of_update', 'time_gone_by')

结果:

df2.show()
+------+-------------------+------------+
|status|timestamp_of_update|time_gone_by|
+------+-------------------+------------+
|    OK|2020-01-01 12:40:00|        null|
|    OK|2020-01-01 13:00:00|        20.0|
|    OK|2020-01-01 13:15:00|        15.0|
|Broken|2020-01-01 13:35:00|        20.0|
|Broken|2020-01-01 13:40:00|        25.0|
|Broken|2020-01-01 14:00:00|        45.0|
|Broken|2020-01-01 14:10:00|        55.0|
|    OK|2020-01-01 14:15:00|         0.0|
|    OK|2020-01-01 14:30:00|        15.0|
+------+-------------------+------------+

---

幕后：

+------+-------------------+---------+------------+-------------------+-------------------+------------+
|status|timestamp_of_update|first_row|change_to_ok|            last_ok|               time|time_gone_by|
+------+-------------------+---------+------------+-------------------+-------------------+------------+
|    OK|2020-01-01 12:40:00|     true|        null|2020-01-01 12:40:00|               null|        null|
|    OK|2020-01-01 13:00:00|    false|       false|2020-01-01 13:00:00|2020-01-01 12:40:00|        20.0|
|    OK|2020-01-01 13:15:00|    false|       false|2020-01-01 13:15:00|2020-01-01 13:00:00|        15.0|
|Broken|2020-01-01 13:35:00|    false|       false|2020-01-01 13:15:00|2020-01-01 13:15:00|        20.0|
|Broken|2020-01-01 13:40:00|    false|       false|2020-01-01 13:15:00|2020-01-01 13:15:00|        25.0|
|Broken|2020-01-01 14:00:00|    false|       false|2020-01-01 13:15:00|2020-01-01 13:15:00|        45.0|
|Broken|2020-01-01 14:10:00|    false|       false|2020-01-01 13:15:00|2020-01-01 13:15:00|        55.0|
|    OK|2020-01-01 14:15:00|    false|        true|2020-01-01 14:15:00|2020-01-01 14:15:00|         0.0|
|    OK|2020-01-01 14:30:00|    false|       false|2020-01-01 14:30:00|2020-01-01 14:15:00|        15.0|
+------+-------------------+---------+------------+-------------------+-------------------+------------+