Scala 如何在Spark SQL中对两列进行聚合
现在,我有了一个包含以下任务的表:Scala 如何在Spark SQL中对两列进行聚合,scala,apache-spark,apache-spark-sql,Scala,Apache Spark,Apache Spark Sql,现在,我有了一个包含以下任务的表: 按部门ID和员工ID上的函数分组 在每个组中,我需要按(ArrivalDate、ArrivalTime)订购并选择第一个。因此,如果两个日期不同,请选择较新的日期。如果两个日期相同,请选择较新的时间 我正在尝试这种方法: input.select("DepartmenId","EmolyeeID", "ArrivalDate", "ArrivalTime", "Word") .agg(here will be the function that handl
input.select("DepartmenId","EmolyeeID", "ArrivalDate", "ArrivalTime", "Word")
.agg(here will be the function that handles logic from 2)
.show()
这里要聚合的语法是什么
先谢谢你
//+-----------+---------+-----------+-----------+--------+
//|出发|情绪化|到达日期|到达时间|单词|
// +-----------+---------+-----------+-----------+--------+
//| D1 | E1 | 20170101 | 0730 |“是”|
// +-----------+---------+-----------+-----------+--------+
//| D1 | E1 | 20170102 | 1530 |“否”|
// +-----------+---------+-----------+-----------+--------+
//| D1 | E2 | 20170101 | 0730 |“动物园”|
// +-----------+---------+-----------+-----------+--------+
//| D1 | E2 | 20170102 | 0330 |“BOO”|
// +-----------+---------+-----------+-----------+--------+
//| D2 | E1 | 20170101 | 0730 |“LOL”|
// +-----------+---------+-----------+-----------+--------+
//| D2 | E1 | 20170101 | 1830 |“附件”|
// +-----------+---------+-----------+-----------+--------+
//| D2 | E2 | 20170105 | 1430 |“统一”|
// +-----------+---------+-----------+-----------+--------+
//输出应该是
// +-----------+---------+-----------+-----------+--------+
//|出发|情绪化|到达日期|到达时间|单词|
// +-----------+---------+-----------+-----------+--------+
//| D1 | E1 | 20170102 | 1530 |“否”|
// +-----------+---------+-----------+-----------+--------+
//| D1 | E2 | 20170102 | 0330 |“BOO”|
// +-----------+---------+-----------+-----------+--------+
//| D2 | E1 | 20170101 | 1830 |“附件”|
// +-----------+---------+-----------+-----------+--------+
//| D2 | E2 | 20170105 | 1430 |“统一”|
//+----------------+----------------+------+-----------+-----------+
您可以在包含所有非分组列的新结构列上使用max
,首先使用ArrivalData
,其次使用ArrivalTime
:新列的排序将符合您的要求(首先是最晚的日期;首先是相似日期中的最晚时间)因此,获得最大值将产生您所追求的记录
然后,可以使用select
操作将结构“拆分”回单独的列
import spark.implicits._
import org.apache.spark.sql.functions._
df.groupBy($"DepartmentID", $"EmployeeID")
.agg(max(struct("ArrivalDate", "ArrivalTime", "Word")) as "struct")
.select($"DepartmentID", $"EmployeeID",
$"struct.ArrivalDate" as "ArrivalDate",
$"struct.ArrivalTime" as "ArrivalTime",
$"struct.Word" as "Word"
)
您可以在包含所有非分组列的新结构列上使用
max
,首先是ArrivalData
,其次是ArrivalTime
:新列的排序将符合您的要求(最晚日期优先;类似日期中的最晚时间优先)因此,获得最大值将产生您想要的记录
然后,可以使用select
操作将结构“拆分”回单独的列
import spark.implicits._
import org.apache.spark.sql.functions._
df.groupBy($"DepartmentID", $"EmployeeID")
.agg(max(struct("ArrivalDate", "ArrivalTime", "Word")) as "struct")
.select($"DepartmentID", $"EmployeeID",
$"struct.ArrivalDate" as "ArrivalDate",
$"struct.ArrivalTime" as "ArrivalTime",
$"struct.Word" as "Word"
)
一种方法是使用火花窗功能:
val df = Seq(
("D1", "E1", "20170101", "0730", "YES"),
("D1", "E1", "20170102", "1530", "NO"),
("D1", "E2", "20170101", "0730", "ZOO"),
("D1", "E2", "20170102", "0330", "BOO"),
("D2", "E1", "20170101", "0730", "LOL"),
("D2", "E1", "20170101", "1830", "ATT"),
("D2", "E2", "20170105", "1430", "UNI")
).toDF(
"DepartmenId", "EmolyeeID", "ArrivalDate", "ArrivalTime", "Word"
)
import org.apache.spark.sql.expressions.Window
val df2 = df.withColumn("rowNum", row_number().over(
Window.partitionBy("DepartmenId", "EmolyeeID").
orderBy($"ArrivalDate".desc, $"ArrivalTime".desc)
)).
select("DepartmenId", "EmolyeeID", "ArrivalDate", "ArrivalTime","Word").
where($"rowNum" === 1).
orderBy("DepartmenId", "EmolyeeID")
df2.show
+-----------+---------+-----------+-----------+----+
|DepartmenId|EmolyeeID|ArrivalDate|ArrivalTime|Word|
+-----------+---------+-----------+-----------+----+
| D1| E1| 20170102| 1530| NO|
| D1| E2| 20170102| 0330| BOO|
| D2| E1| 20170101| 1830| ATT|
| D2| E2| 20170105| 1430| UNI|
+-----------+---------+-----------+-----------+----+
一种方法是使用火花窗功能:
val df = Seq(
("D1", "E1", "20170101", "0730", "YES"),
("D1", "E1", "20170102", "1530", "NO"),
("D1", "E2", "20170101", "0730", "ZOO"),
("D1", "E2", "20170102", "0330", "BOO"),
("D2", "E1", "20170101", "0730", "LOL"),
("D2", "E1", "20170101", "1830", "ATT"),
("D2", "E2", "20170105", "1430", "UNI")
).toDF(
"DepartmenId", "EmolyeeID", "ArrivalDate", "ArrivalTime", "Word"
)
import org.apache.spark.sql.expressions.Window
val df2 = df.withColumn("rowNum", row_number().over(
Window.partitionBy("DepartmenId", "EmolyeeID").
orderBy($"ArrivalDate".desc, $"ArrivalTime".desc)
)).
select("DepartmenId", "EmolyeeID", "ArrivalDate", "ArrivalTime","Word").
where($"rowNum" === 1).
orderBy("DepartmenId", "EmolyeeID")
df2.show
+-----------+---------+-----------+-----------+----+
|DepartmenId|EmolyeeID|ArrivalDate|ArrivalTime|Word|
+-----------+---------+-----------+-----------+----+
| D1| E1| 20170102| 1530| NO|
| D1| E2| 20170102| 0330| BOO|
| D2| E1| 20170101| 1830| ATT|
| D2| E2| 20170105| 1430| UNI|
+-----------+---------+-----------+-----------+----+
这是可行的!但是有没有一种方法可以避免在struct()中出现“Word”?因为在我的真实示例中,“Word”不止一列。因此,图像中有“Word1”、“Word2”、“Word3”…,有没有一种方法可以让我只在.select语句中添加这些词?只有在“组合”时,这种方法才有效将您感兴趣的所有列合并到一个结构中-否则,仅对某些列使用
max
将无法为其他列获取正确的值。如果您有许多这样的列,您可以尝试@leo-c建议的替代窗口方法,或者对df.columns
使用一些筛选来动态生成结构中包含的列的列表,这样您就不必显式地列出它们。这是可行的!但是有什么方法可以避免在结构()中包含“Word”?因为在我的真实示例中,“Word”不止一列。所以图像中有“Word1”、“Word2”、“Word3”…,是否有一种方法可以将这些单词添加到.select语句中?只有在“组合”时,这种方法才有效将您感兴趣的所有列合并到一个结构中-否则,仅对某些列使用max
将无法为其他列获取正确的值。如果您有许多这样的列,您可以尝试@leo-c建议的替代窗口方法,或者对df.columns
使用一些筛选来动态生成结构中包含的列列表,这样您就不必显式地列出它们。谢谢!使用Window.Partition比使用GroupBy好吗?我想知道在速度和性能上是否有折衷performance@vincwng,我没有对窗口函数与groupBy进行任何基准测试,但我发现窗口函数在降低必要的嵌套查询(例如自联接)在这种情况下,他们会提供更好的性能。谢谢!使用Window.Partition比使用GroupBy好吗?我想知道在速度和性能上是否有折衷performance@vincwng,我没有对窗口函数与groupBy进行任何基准测试,但我发现窗口函数在降低必要的嵌套查询级别方面通常很有用(例如,自联接)在这种情况下,它们将提供更好的性能。