使用Spark SQL获取每小时更新

我正在处理一个orders表，该表包含order\u id和order\u completion\u date字段。订单完成日期显示特定订单完成的时间。如果订单未完成，则会显示4447-12-30 00:00:00这样的值

我的目标是在过去24小时内获得每小时完成的记录数。我研究了其他类似的问题，但找不到用以下格式表示时间戳的方法：

       {
           "date_time":"07/05/2018 00:00:00", 
           "completed_orders":"1000" 
        },
        {
           "date_time":"07/05/2018 01:00:00",
           "completed_orders":"2000"
        },
        {
           "date_time":"07/05/2018 02:00:00",
           "completed_orders":"3000"
        },
        {
           "date_time":"07/05/2018 03:00:00",
           "completed_orders":"4000"
        },
        {
           "date_time":"07/05/2018 04:00:00",
           "completed_orders":"5000"
        }

我正在尝试以下查询：

sqlContext.sql("SELECT COUNT(csg_order_id) AS completed_orders, hour(from_unixtime(cast(ord_completion_date as bigint))) as hour "
            + "FROM TotalTasks "
            + "WHERE ord_completion_date <= cast('"+curDate+"' as Timestamp) AND ord_completion_date >= date_add(cast('"+curDate+ "' as Timestamp), -1) "
            + "GROUP BY  hour "
            + "ORDER BY  hour");

有没有什么方法可以得到时间而不是小时数？非常感谢您的帮助。

您可以使用SQL函数：

SELECT
  COUNT(*) AS completed,
  date_format(from_unixtime(bigint(ord_completion_date))), 'MM/dd/yyyy HH:00:00') AS date_time
 FROM tasks
 WHERE bigint(ord_completion_date) BETWEEN current_date AND date_add(current_date, 1)
 GROUP BY 2
 ORDER BY 2

---

但它将按给定的日期和时间格式分组，而不是按小时分组。我想将过去24小时内所有ord_完成日期的记录按小时分组。按小时分组和按格式化日期分组在逻辑上没有区别。它只会影响性能。但是它会给出0条记录。那么好吧。请提供输入数据集的模式和几个示例。消除误解，以便说。

SELECT
  COUNT(*) AS completed,
  date_format(from_unixtime(bigint(ord_completion_date))), 'MM/dd/yyyy HH:00:00') AS date_time
 FROM tasks
 WHERE bigint(ord_completion_date) BETWEEN current_date AND date_add(current_date, 1)
 GROUP BY 2
 ORDER BY 2