Apache spark 为什么在reduceByKey之后所有数据都会在一个分区中结束?
我有一个简单的spark程序。我想知道为什么所有数据最终都在一个分区中Apache spark 为什么在reduceByKey之后所有数据都会在一个分区中结束?,apache-spark,rdd,Apache Spark,Rdd,我有一个简单的spark程序。我想知道为什么所有数据最终都在一个分区中 val l = List((30002,30000), (50006,50000), (80006,80000), (4,0), (60012,60000), (70006,70000), (40006,40000), (30012,30000), (30000,30000), (60018,60000), (30020,30000), (20
val l = List((30002,30000), (50006,50000), (80006,80000),
(4,0), (60012,60000), (70006,70000),
(40006,40000), (30012,30000), (30000,30000),
(60018,60000), (30020,30000), (20010,20000),
(20014,20000), (90008,90000), (14,0), (90012,90000),
(50010,50000), (100008,100000), (80012,80000),
(20000,20000), (30010,30000), (20012,20000),
(90016,90000), (18,0), (12,0), (70016,70000),
(20,0), (80020,80000), (100016,100000), (70014,70000),
(60002,60000), (40000,40000), (60006,60000),
(80000,80000), (50008,50000), (60008,60000),
(10002,10000), (30014,30000), (70002,70000),
(40010,40000), (100010,100000), (40002,40000),
(20004,20000),
(10018,10000), (50018,50000), (70004,70000),
(90004,90000), (100004,100000), (20016,20000))
val l_rdd = sc.parallelize(l, 2)
// print each item and index of the partition it belongs to
l_rdd.mapPartitionsWithIndex((index, iter) => {
iter.toList.map(x => (index, x)).iterator
}).collect.foreach(println)
// reduce on the second element of the list.
// alternatively you can use aggregateByKey
val l_reduced = l_rdd.map(x => {
(x._2, List(x._1))
}).reduceByKey((a, b) => {b ::: a})
// print the reduced results along with its partition index
l_reduced.mapPartitionsWithIndex((index, iter) => {
iter.toList.map(x => (index, x._1, x._2.size)).iterator
}).collect.foreach(println)
当您运行此程序时,您将看到数据(
l_rdd
)分布到两个分区中。一旦我缩减,结果RDD(l_reduced
)也有两个分区,但所有数据都在一个分区(索引0)中,而另一个分区为空。即使数据量很大(几GB),也会发生这种情况。l_reduced
不也应该分配到两个分区中吗 除非您另外指定,否则分区将基于相关键的hashcode完成,并假设hashcode将导致相对均匀的分布。在本例中,您的哈希代码都是偶数,因此都将进入分区0
如果这确实代表了您的数据集,那么reduceByKey
会有一个重载,它会使用分区器和reduce函数。我建议为这样的数据集提供一种替代的分区算法
val l_reduced = l_rdd.map(x => {
(x._2, List(x._1))
}).reduceByKey((a, b) => {b ::: a})
参考上面的代码片段,您正在按照RDD的第二个字段进行分区。第二个字段中的所有数字都以0结尾
调用HashPartitioner时,记录的分区号由以下内容决定:
而Utils.nonNegativeMod的定义如下:
因此,您的所有记录最终都位于分区0中
您可以通过重新分区来解决此问题:
val l_reduced = l_rdd.map(x => {
(x._2, List(x._1))
}).reduceByKey((a, b) => {b ::: a}).repartition(2)
其中:
(0,100000,4)
(0,10000,2)
(0,0,5)
(0,20000,6)
(0,60000,5)
(0,80000,4)
(1,50000,4)
(1,30000,6)
(1,90000,4)
(1,70000,5)
(1,40000,4)
或者,您可以创建一个
scala> l.map(_._2.hashCode % 2) // numPartitions = 2
res10: List[Int] = List(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
val l_reduced = l_rdd.map(x => {
(x._2, List(x._1))
}).reduceByKey((a, b) => {b ::: a}).repartition(2)
(0,100000,4)
(0,10000,2)
(0,0,5)
(0,20000,6)
(0,60000,5)
(0,80000,4)
(1,50000,4)
(1,30000,6)
(1,90000,4)
(1,70000,5)
(1,40000,4)