Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/scala/16.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
仅从scala中的Epochtime提取小时_Scala_Dataframe_Apache Spark_Apache Spark Sql - Fatal编程技术网

仅从scala中的Epochtime提取小时

仅从scala中的Epochtime提取小时,scala,dataframe,apache-spark,apache-spark-sql,Scala,Dataframe,Apache Spark,Apache Spark Sql,我有一个数据帧,其中一列是epochtime。 我只想从中提取小时,并将其显示为一个单独的列 下面是示例数据帧: +----------+-------------+ | NUM_ID| STIME| +----------+-------------+ |xxxxxxxx01|1571634285000| |xxxxxxxx01|1571634299000| |xxxxxxxx01|1571634311000| |xxxxxxxx01|1571634316000| |xxx

我有一个数据帧,其中一列是epochtime。 我只想从中提取小时,并将其显示为一个单独的列

下面是示例数据帧:

+----------+-------------+
|    NUM_ID|        STIME|
+----------+-------------+
|xxxxxxxx01|1571634285000|
|xxxxxxxx01|1571634299000|
|xxxxxxxx01|1571634311000|
|xxxxxxxx01|1571634316000|
|xxxxxxxx02|1571634318000|
|xxxxxxxx02|1571398176000|
|xxxxxxxx02|1571627596000|
以下是预期产出

+----------+-------------+-----+
|    NUM_ID|        STIME| HOUR|
+----------+-------------+-----+
|xxxxxxxx01|1571634285000|  10 |
|xxxxxxxx01|1571634299000|  10 |
|xxxxxxxx01|1571634311000|  10 |
|xxxxxxxx01|1571634316000|  10 |
|xxxxxxxx02|1571634318000|  10 |
|xxxxxxxx02|1571398176000|  16 |
|xxxxxxxx02|1571627596000|  08 |

我试过了

val test = test1DF.withColumn("TIME", extract HOUR(from_unixtime($"STIME"/1000)))
这会引发异常

<console>:46: error: not found: value extract
数据帧中时间的数据类型很长


从长数据类型的epochtime中提取小时的任何线索?

从时间戳中提取小时与使用以下函数一样简单:

import org.apache.spark.sql.functions._

val df_with_hour = df.withColumn("TIME", hour(from_unixtime($"STIME" / 1000)))

df_with_hour.show()
// +-------------+----+
// |        STIME|TIME|
// +-------------+----+
// |1571634285000|   5|
// |1571398176000|  11|
// |1571627596000|   3|
// +-------------+----+
注意:我在不同的时区

它使用val test=test1DF.withColumnTIME,hourfrom_unixtime$STIME/1000
import org.apache.spark.sql.functions._

val df_with_hour = df.withColumn("TIME", hour(from_unixtime($"STIME" / 1000)))

df_with_hour.show()
// +-------------+----+
// |        STIME|TIME|
// +-------------+----+
// |1571634285000|   5|
// |1571398176000|  11|
// |1571627596000|   3|
// +-------------+----+