Python Pyspark dataframe将函数应用于一行,并将行添加到dataframe的底部

Python Pyspark dataframe将函数应用于一行,并将行添加到dataframe的底部,python,pyspark,Python,Pyspark,我有一个只有一行的df id |id2 |score|score2| ---------------------- 0 |1 |4 |2 | 我想在底部加一行百分比,即每个数字除以7 0/7 |1/7 |4/7 |2/7 | 但我想出的解决方案非常缓慢 temp = [i/7 for i in df.collect()[0]] row = sc.parallelize(Row(temp)).toDF() df.union(row) 这花了21秒

我有一个只有一行的df

id   |id2  |score|score2|
----------------------
0    |1    |4    |2     |
我想在底部加一行百分比,即每个数字除以7

0/7  |1/7  |4/7  |2/7   |

但我想出的解决方案非常缓慢

temp = [i/7 for i in df.collect()[0]]
row = sc.parallelize(Row(temp)).toDF()
df.union(row)


这花了21秒运行,几乎所有的代码都是最后两行代码。有更好的方法吗?我的另一个想法是转置表,然后使用df.withColumn()就可以轻松完成。理想情况下,我还想用0过滤掉该列,但我还没有真正研究过这一点

 from pyspark.sql import SparkSession
 from pyspark.sql import functions as F

 spark = SparkSession.builder \
.appName('practice')\
.getOrCreate()

 sc= spark.sparkContext

 df = sc.parallelize([
(0,1,4,2)]).toDF(["id", "id2","score","score2"])


df2 = df.select(*[(F.col(column)/7).alias(column) for column in df.columns])

df3 = df.union(df2)

df3.show()
+---+-------------------+------------------+------------------+
| id|                id2|             score|            score2|
+---+-------------------+------------------+------------------+
|0.0|                1.0|               4.0|               2.0|
|0.0|0.14285714285714285|0.5714285714285714|0.2857142857142857|
+---+-------------------+------------------+------------------+
如果你想的话。过滤掉包含0的列,您可以使用下面的代码

non_zero_cols  = [c for c in df.columns if df[[c]].first()[c] > 0]

df1 = df.select(*non_zero_cols)

df2 = df1.select(*[(F.col(column)/7).alias(column) for column in 
df1.columns])

df3 = df1.union(df2)

df3.show()

+-------------------+------------------+------------------+
|                id2|             score|            score2|
+-------------------+------------------+------------------+
|                1.0|               4.0|               2.0|
|0.14285714285714285|0.5714285714285714|0.2857142857142857|
+-------------------+------------------+------------------+
请检查以下df类型列的代码

non_zero_cols  = [c for c in df.columns if df[[c]].first()[c] > 0]

df1 = df.select(*non_zero_cols, F.lit('count').alias('type') )

df2 = df1.select(*[(F.col(column)/7).alias(column) for column in 
df1.columns if not column=='type'], F.lit('percent').alias('type'))

df3 = df1.union(df2)

df3.show()

+-------------------+------------------+------------------+-------+
|                id2|             score|            score2|   type|
+-------------------+------------------+------------------+-------+
|                1.0|               4.0|               2.0|  count|
|0.14285714285714285|0.5714285714285714|0.2857142857142857|percent|
+-------------------+------------------+------------------+-------+

非常感谢。这正是我要找的!有没有办法添加一列来说明每一行代表什么?是否像列“type”中有“count”、“percent”等条目?