用java中的Spark数据帧在数据库中持久化窗口函数的输出
执行以下代码段时:用java中的Spark数据帧在数据库中持久化窗口函数的输出,java,scala,apache-spark,pyspark,apache-spark-sql,Java,Scala,Apache Spark,Pyspark,Apache Spark Sql,执行以下代码段时: Dataset<Row> ds1=ds.groupBy(functions.window(ds.col("datetime"),windowLength,slidingLength).as("datetime"),ds.col("symbol").as("Ticker")) .agg(functions.mean("volume").as("volume"),functions.mean("price").as("Price"),
Dataset<Row> ds1=ds.groupBy(functions.window(ds.col("datetime"),windowLength,slidingLength).as("datetime"),ds.col("symbol").as("Ticker"))
.agg(functions.mean("volume").as("volume"),functions.mean("price").as("Price"),
(functions.first("price").plus(functions.last("price")).divide(value)).as("Mid_Point"),
functions.max("price").as("High"),functions.min("price").as("Low"),
functions.first("price").as("Open"),functions.last("price").as("Close"))
.sort(functions.asc("datetime"));
ds1.printSchema();
现在,当我试图将其保存到csv文件中时,我得到一个错误,csv文件无法将日期时间解析为时间戳
错误:
cannot resolve 'CAST(`datetime` AS TIMESTAMP)' due to data type mismatch: cannot cast StructType(StructField(start,TimestampType,true), StructField(end,TimestampType,true)) to TimestampType
有人知道吗?将日期时间转换应用于列,而不是应用于滑动窗口
ds.col("datetime").as("datetime")
datetime不是强制转换类型,而是列名。对于强制转换,我使用以下代码:ds.withColumn(“datetime”,ds.col(“datetime”).cast(“timestamp”))
ds.col("datetime").as("datetime")