Pyspark通过列表理解从datetime获取时间属性
我有一个pyspark数据帧df:Pyspark通过列表理解从datetime获取时间属性,datetime,pyspark,python-datetime,pyspark-dataframes,Datetime,Pyspark,Python Datetime,Pyspark Dataframes,我有一个pyspark数据帧df: +-------------------+ | timestamplast| +-------------------+ |2019-08-01 00:00:00| |2019-08-01 00:01:09| |2019-08-01 01:00:20| |2019-08-03 00:00:27| +-------------------+ 我想通过列表理解将“年”、“月”、“日”、“小时”列添加到现有的数据框架中 在熊猫中,应这样做: L = ['
+-------------------+
| timestamplast|
+-------------------+
|2019-08-01 00:00:00|
|2019-08-01 00:01:09|
|2019-08-01 01:00:20|
|2019-08-03 00:00:27|
+-------------------+
我想通过列表理解将“年”、“月”、“日”、“小时”列添加到现有的数据框架中
在熊猫中,应这样做:
L = ['year', 'month', 'day', 'hour']
date_gen = (getattr(df['timestamplast'].dt, i).rename(i) for i in L)
df = df.join(pd.concat(date_gen, axis=1)) # concatenate results and join to original dataframe
在pyspark中如何执行此操作?请检查以下内容:
df.selectExpr("*", *[ '{0}(timestamplast) as {0}'.format(c) for c in L]).show()
+-------------------+----+-----+---+----+
| timestamplast|year|month|day|hour|
+-------------------+----+-----+---+----+
|2019-08-01 00:00:00|2019| 8| 1| 0|
|2019-08-03 00:00:27|2019| 8| 3| 0|
+-------------------+----+-----+---+----+