Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Tsql DATEADD的Pyspark实现_Tsql_Apache Spark_Pyspark_Pyspark Sql - Fatal编程技术网

Tsql DATEADD的Pyspark实现

Tsql DATEADD的Pyspark实现,tsql,apache-spark,pyspark,pyspark-sql,Tsql,Apache Spark,Pyspark,Pyspark Sql,我的T-SQL代码如下所示 cast( dateadd(minute, - 240, tmp_view_tos_lenelgate_qry11.eventdate) as date 如何在PYSPARK中实现添加日期功能 # Creating the DataFrame df = spark.createDataFrame([('2014-02-13 12:36:52.721',),('2018-01-01 00:30:50.001',)], ['eventdate']) df = df.wi

我的T-SQL代码如下所示

cast( dateadd(minute, - 240, tmp_view_tos_lenelgate_qry11.eventdate) as date
如何在PYSPARK中实现添加日期功能

# Creating the DataFrame
df = spark.createDataFrame([('2014-02-13 12:36:52.721',),('2018-01-01 00:30:50.001',)], ['eventdate'])
df = df.withColumn('eventdate', col('eventdate').cast('timestamp'))
df.show(truncate=False)
+-----------------------+
|eventdate              |
+-----------------------+
|2014-02-13 12:36:52.721|
|2018-01-01 00:30:50.001|
+-----------------------+
df.printSchema()
root
 |-- eventdate: timestamp (nullable = true)

# Subtract 240 minutes/240*60=14400 seconds from 'eventdate'
from pyspark.sql.functions import col, unix_timestamp
df = df.withColumn('eventdate_new', (unix_timestamp('eventdate') - 240*60).cast('timestamp'))
df.show(truncate=False)
+-----------------------+-------------------+
|eventdate              |eventdate_new      |
+-----------------------+-------------------+
|2014-02-13 12:36:52.721|2014-02-13 08:36:52|
|2018-01-01 00:30:50.001|2017-12-31 20:30:50|
+-----------------------+-------------------+