Apache spark 更改分区大小是否应该影响查询的输出？

我正在练习一本关于火花的书中的一些例子。在其中的一个示例中，我从

.csv

文件中读取了一些数据

val staticDataFrame = spark.read.format("csv")
.option("header", "true")
.option("inferSchema", "true")
.load("/data/retail-data/by-day/*.csv")

然后创建一个

sql

视图

staticDataFrame.createOrReplaceTempView("retail_data")
val staticSchema = staticDataFrame.schema

然后运行一个查询

import org.apache.spark.sql.functions.{window, column, desc, col}
staticDataFrame
.selectExpr(
"CustomerId",
"(UnitPrice * Quantity) as total_cost",
"InvoiceDate")
.groupBy(
col("CustomerId"), window(col("InvoiceDate"), "1 day"))
.sum("total_cost")
.show(5)

我得到以下输出

+----------+--------------------+-----------------+
|CustomerId|              window|  sum(total_cost)|
+----------+--------------------+-----------------+
|   16057.0|[2011-12-05 00:00...|            -37.6|
|   14126.0|[2011-11-29 00:00...|643.6300000000001|
|   13500.0|[2011-11-16 00:00...|497.9700000000001|
|   17160.0|[2011-11-08 00:00...|516.8499999999999|
|   15608.0|[2011-11-11 00:00...|            122.4|
+----------+--------------------+-----------------+

然后我改变了分区大小，再次运行相同的查询。但是我得到了不同的输出

    scala> spark.conf.set("spark.sql.shuffle.partitions","5");

scala> staticDataFrame.
     | selectExpr(
     | "CustomerId",
     | "(UnitPrice * Quantity) as total_cost",
     | "InvoiceDate").
     | groupBy(
     | col("CustomerId"),window(col("InvoiceDate"),"1 day")).
     | sum("total_cost").
     | show(5)


+----------+--------------------+------------------+
|CustomerId|              window|   sum(total_cost)|
+----------+--------------------+------------------+
|   14075.0|[2011-12-05 00:00...|316.78000000000003|
|   18180.0|[2011-12-05 00:00...|            310.73|
|   15358.0|[2011-12-05 00:00...| 830.0600000000003|
|   15392.0|[2011-12-05 00:00...|304.40999999999997|
|   15290.0|[2011-12-05 00:00...|263.02000000000004|
+----------+--------------------+------------------+
only showing top 5 rows

---

这是预期的行为。两种情况下的输出是否相同？

您的数据帧有多少条记录？不过没关系

我相信它的行为和预期的一样，因为您只显示了5条记录，您的第二个查询在分区后返回了不同的数据集

尝试对某些列进行排序，并获得前5个结果，它应该在分区之前和之后为您提供相同的结果

谢谢