仅从scala中的Epochtime提取小时
我有一个数据帧,其中一列是epochtime。 我只想从中提取小时,并将其显示为一个单独的列 下面是示例数据帧:仅从scala中的Epochtime提取小时,scala,dataframe,apache-spark,apache-spark-sql,Scala,Dataframe,Apache Spark,Apache Spark Sql,我有一个数据帧,其中一列是epochtime。 我只想从中提取小时,并将其显示为一个单独的列 下面是示例数据帧: +----------+-------------+ | NUM_ID| STIME| +----------+-------------+ |xxxxxxxx01|1571634285000| |xxxxxxxx01|1571634299000| |xxxxxxxx01|1571634311000| |xxxxxxxx01|1571634316000| |xxx
+----------+-------------+
| NUM_ID| STIME|
+----------+-------------+
|xxxxxxxx01|1571634285000|
|xxxxxxxx01|1571634299000|
|xxxxxxxx01|1571634311000|
|xxxxxxxx01|1571634316000|
|xxxxxxxx02|1571634318000|
|xxxxxxxx02|1571398176000|
|xxxxxxxx02|1571627596000|
以下是预期产出
+----------+-------------+-----+
| NUM_ID| STIME| HOUR|
+----------+-------------+-----+
|xxxxxxxx01|1571634285000| 10 |
|xxxxxxxx01|1571634299000| 10 |
|xxxxxxxx01|1571634311000| 10 |
|xxxxxxxx01|1571634316000| 10 |
|xxxxxxxx02|1571634318000| 10 |
|xxxxxxxx02|1571398176000| 16 |
|xxxxxxxx02|1571627596000| 08 |
我试过了
val test = test1DF.withColumn("TIME", extract HOUR(from_unixtime($"STIME"/1000)))
这会引发异常
<console>:46: error: not found: value extract
数据帧中时间的数据类型很长
从长数据类型的epochtime中提取小时的任何线索?从时间戳中提取小时与使用以下函数一样简单:
import org.apache.spark.sql.functions._
val df_with_hour = df.withColumn("TIME", hour(from_unixtime($"STIME" / 1000)))
df_with_hour.show()
// +-------------+----+
// | STIME|TIME|
// +-------------+----+
// |1571634285000| 5|
// |1571398176000| 11|
// |1571627596000| 3|
// +-------------+----+
注意:我在不同的时区它使用val test=test1DF.withColumnTIME,hourfrom_unixtime$STIME/1000
import org.apache.spark.sql.functions._
val df_with_hour = df.withColumn("TIME", hour(from_unixtime($"STIME" / 1000)))
df_with_hour.show()
// +-------------+----+
// | STIME|TIME|
// +-------------+----+
// |1571634285000| 5|
// |1571398176000| 11|
// |1571627596000| 3|
// +-------------+----+