Arrays 如何将结构或类的数组从UDF返回到dataframe列值中?
我只想将列值设置为UDF返回的stuct数组。它给我的错误是: TypeError:new()正好接受3个参数(给定1个) TypeError回溯(最近的调用 最后)在() 22返回日期 23 --->24 MergeAdjacentUsages=udf(MergeAdjacentUsage,ArrayType(Dates())) 25 26 df1=df.groupBy(['ID','pID']).agg(MergeAdjacentUsages(F.collect_list(struct('startTime','endTime')))。别名(“Times”))Arrays 如何将结构或类的数组从UDF返回到dataframe列值中?,arrays,dataframe,struct,pyspark,user-defined-functions,Arrays,Dataframe,Struct,Pyspark,User Defined Functions,我只想将列值设置为UDF返回的stuct数组。它给我的错误是: TypeError:new()正好接受3个参数(给定1个) TypeError回溯(最近的调用 最后)在() 22返回日期 23 --->24 MergeAdjacentUsages=udf(MergeAdjacentUsage,ArrayType(Dates())) 25 26 df1=df.groupBy(['ID','pID']).agg(MergeAdjacentUsages(F.collect_list(struct('s
任何帮助、想法或提示都将不胜感激。pyspark不允许用户定义类对象作为数据框列类型。相反,我们需要创建
StructType
,它可以类似于python中的类/命名元组
例如:
d = [{'ID': '1', 'pID': 1000, 'startTime':'2018.07.02T03:34:20', 'endTime':'2018.07.03T02:40:20'}, {'ID': '1', 'pID': 1000, 'startTime':'2018.07.02T03:45:20', 'endTime':'2018.07.03T02:50:20'}, {'ID': '2', 'pID': 2000, 'startTime':'2018.07.02T03:34:20', 'endTime':'2018.07.03T02:40:20'}, {'ID': '2', 'pID': 2000, 'startTime':'2018.07.02T03:45:20', 'endTime':'2018.07.03T02:50:20'}]
df = spark.createDataFrame(d)
Dates = namedtuple("Dates", "startTime endTime")
def MergeAdjacentUsage(timeSets):
DatesArray = []
for times in timeSets:
DatesArray.append(Dates(startTime=times.startTime, endTime=times.endTime))
return DatesArray
MergeAdjacentUsages = udf(MergeAdjacentUsage,ArrayType(Dates()))
df1=df.groupBy(['ID','pID']).agg(MergeAdjacentUsages(F.collect_list(struct('startTime','endTime'))).alias("Times"))
display(df1)
希望这有帮助
from pyspark.sql.types import *
from pyspark.sql.functions import udf
from pyspark.sql import functions as F
# from pyspark.sql.functions import *
d = [{'ID': '1', 'pID': 1000, 'startTime': '2018.07.02T03:34:20', 'endTime': '2018.07.03T02:40:20'},
{'ID': '1', 'pID': 1000, 'startTime': '2018.07.02T03:45:20', 'endTime': '2018.07.03T02:50:20'},
{'ID': '2', 'pID': 2000, 'startTime': '2018.07.02T03:34:20', 'endTime': '2018.07.03T02:40:20'},
{'ID': '2', 'pID': 2000, 'startTime': '2018.07.02T03:45:20', 'endTime': '2018.07.03T02:50:20'}]
df = spark.createDataFrame(d)
# Dates = namedtuple("Dates", "startTime endTime")
schema = ArrayType(StructType([
StructField("startTime", StringType(), False),
StructField("endTime", StringType(), False)
]))
MergeAdjacentUsages = udf(lambda xs: xs, schema)
df1 = df.groupBy(['ID', 'pID']).agg(MergeAdjacentUsages(
F.collect_list(F.struct('startTime', 'endTime'))).alias("Times"))
df1.show(truncate=False)
+---+----+----------------------------------------------------------------------------------------+
|ID |pID |Times |
+---+----+----------------------------------------------------------------------------------------+
|2 |2000|[[2018.07.02T03:34:20, 2018.07.03T02:40:20], [2018.07.02T03:45:20, 2018.07.03T02:50:20]]|
|1 |1000|[[2018.07.02T03:34:20, 2018.07.03T02:40:20], [2018.07.02T03:45:20, 2018.07.03T02:50:20]]|
+---+----+----------------------------------------------------------------------------------------+