Scala 火花聚合器工作时遇到问题
我想在Scala Spark中试用聚合器,但我似乎无法让它们同时使用Scala 火花聚合器工作时遇到问题,scala,apache-spark,apache-spark-sql,aggregate-functions,user-defined-functions,Scala,Apache Spark,Apache Spark Sql,Aggregate Functions,User Defined Functions,我想在Scala Spark中试用聚合器,但我似乎无法让它们同时使用select函数和groupBy/agg函数工作(在我当前的实现中,agg函数无法编译)。我的聚合器写在下面,应该是不言自明的 import org.apache.spark.sql.expressions.Aggregator import org.apache.spark.sql.{Encoder, Encoders} /** Stores the number of true counts (tc) and false
select
函数和groupBy/agg
函数工作(在我当前的实现中,agg
函数无法编译)。我的聚合器写在下面,应该是不言自明的
import org.apache.spark.sql.expressions.Aggregator
import org.apache.spark.sql.{Encoder, Encoders}
/** Stores the number of true counts (tc) and false counts (fc) */
case class Counts(var tc: Long, var fc: Long)
/** Count the number of true and false occurances of a function */
class BooleanCounter[A](f: A => Boolean) extends Aggregator[A, Counts, Counts] with Serializable {
// Initialize both counts to zero
def zero: Counts = Counts(0L, 0L)
// Sum counts for intermediate value and new value
def reduce(acc: Counts, other: A): Counts = {
if (f(other)) acc.tc += 1 else acc.fc += 1
acc
}
// Sum counts for intermediate values
def merge(acc1: Counts, acc2: Counts): Counts = {
acc1.tc += acc2.tc
acc1.fc += acc2.fc
acc1
}
// Return results
def finish(acc: Counts): Counts = acc
// Encoder for intermediate value type
def bufferEncoder: Encoder[Counts] = Encoders.product[Counts]
// Encoder for return type
def outputEncoder: Encoder[Counts] = Encoders.product[Counts]
}
下面是我的测试代码
val ds: Dataset[Employee] = Seq(
Employee("John", 110),
Employee("Paul", 100),
Employee("George", 0),
Employee("Ringo", 80)
).toDS()
val salaryCounter = new BooleanCounter[Employee]((r: Employee) => r.salary < 10).toColumn
// Usage works fine
ds.select(salaryCounter).show()
// Causes an error
ds.groupBy($"name").agg(salaryCounter).show()
Databricks有一个相当复杂的,但似乎是Spark 2.3。还有一个较旧的教程使用了Spark 1.6中的实验功能 您错误地混合了“静态类型”和“动态类型”API。要使用前一个版本,您应该在KeyValueGroupedDataset
上调用agg
,而不是RelationalGroupedDataset
:
ds.groupByKey(_.name).agg(salaryCounter)
操作:
ds.groupBy($“name”).agg(salaryCounter).show()
?早期聚合的输出只返回一个包含2列(tc、fc)和1行的数据集。此操作的期望输出是什么?当应用于ds.groupBy($“name”)
时,UDAF显然不起作用,因为在这种情况下提供给UDAF的输入不是Employee
,这将理想地生成一个数据集,其中每行对应一个名称,列(tc,fc)每个姓名对应的员工人数分别低于或高于10美元。UDAF不会像您所说的那样应用于ds.groupBy($“name”)
,而是传递给.agg
函数。请参阅我链接的教程,因为它们有一些似乎有效的示例用法。
ds.groupByKey(_.name).agg(salaryCounter)