Scala 我如何解释ApacheSpark RDD沿袭图？

我对以下代码没有什么疑问：

val input1 = rawinput.map(_.split("\t")).map(x=>(x(6).trim(),x)).sortByKey()
val input2 = input1.map(x=> x._2.mkString("\t"))
val x0 = input2.map(_.split("\t")).map(x => (x(6),x(0))
val x1 = input2.map(_.split("\t")).map(x => (x(6),x(1))
val x2 = input2.map(_.split("\t")).map(x => (x(6),x(2))
val x3 = input2.map(_.split("\t")).map(x => (x(6),x(3))
val x4 = input2.map(_.split("\t")).map(x => (x(6),x(4))
val x5 = input2.map(_.split("\t")).map(x => (x(6),x(5))
val x6 = input2.map(_.split("\t")).map(x => (x(6),x(6))
val x = x0 union x1 union x2 union  x3 union x4 union x5 union x6


<pre>
**Lineage Graph:**
(7) UnionRDD[25] at union at rddCustUtil.scala:78 []
|  UnionRDD[24] at union at rddCustUtil.scala:78 []
|  UnionRDD[23] at union at rddCustUtil.scala:78 []
|  UnionRDD[22] at union at rddCustUtil.scala:78 []
|  UnionRDD[21] at union at rddCustUtil.scala:78 []
|  UnionRDD[20] at union at rddCustUtil.scala:78 []
|  MapPartitionsRDD[7] at map at rddCustUtil.scala:43 []
|  MapPartitionsRDD[6] at map at rddCustUtil.scala:43 []
|  MapPartitionsRDD[5] at map at rddCustUtil.scala:40 []
|  ShuffledRDD[4] at sortByKey at rddCustUtil.scala:38 []
+-(1) MapPartitionsRDD[3] at map at rddCustUtil.scala:38 []
    |  MapPartitionsRDD[2] at map at rddCustUtil.scala:38 []
    |  /Data/ MapPartitionsRDD[1] at textFile at rddCustUtil.scala:35 []
    |  /Data/ HadoopRDD[0] at textFile at rddCustUtil.scala:35 []
|  MapPartitionsRDD[9] at map at rddCustUtil.scala:48 []
|  MapPartitionsRDD[8] at map at rddCustUtil.scala:48 []
|  MapPartitionsRDD[5] at map at rddCustUtil.scala:40 []
|  ShuffledRDD[4] at sortByKey at rddCustUtil.scala:38 []
+-(1) MapPartitionsRDD[3] at map at rddCustUtil.scala:38 []
    |  MapPartitionsRDD[2] at map at rddCustUtil.scala:38 []
    |  /Data/ MapPartitionsRDD[1] at textFile at rddCustUtil.scala:35 []
    |  /Data/ HadoopRDD[0] at textFile at rddCustUtil.scala:35 []
|  MapPartitionsRDD[11] at map at rddCustUtil.scala:53 []
|  MapPartitionsRDD[10] at map at rddCustUtil.scala:53 []
|  MapPartitionsRDD[5] at map at rddCustUtil.scala:40 []
|  ShuffledRDD[4] at sortByKey at rddCustUtil.scala:38 []
+-(1) MapPartitionsRDD[3] at map at rddCustUtil.scala:38 []
    |  MapPartitionsRDD[2] at map at rddCustUtil.scala:38 []
    |  /Data/ MapPartitionsRDD[1] at textFile at rddCustUtil.scala:35 []
    |  /Data/ HadoopRDD[0] at textFile at rddCustUtil.scala:35 []
|  MapPartitionsRDD[13] at map at rddCustUtil.scala:58 []
|  MapPartitionsRDD[12] at map at rddCustUtil.scala:58 []
|  MapPartitionsRDD[5] at map at rddCustUtil.scala:40 []
|  ShuffledRDD[4] at sortByKey at rddCustUtil.scala:38 []
+-(1) MapPartitionsRDD[3] at map at rddCustUtil.scala:38 []
    |  MapPartitionsRDD[2] at map at rddCustUtil.scala:38 []
    |  /Data/ MapPartitionsRDD[1] at textFile at rddCustUtil.scala:35 []
    |  /Data/ HadoopRDD[0] at textFile at rddCustUtil.scala:35 []
|  MapPartitionsRDD[15] at map at rddCustUtil.scala:63 []
|  MapPartitionsRDD[14] at map at rddCustUtil.scala:63 []
|  MapPartitionsRDD[5] at map at rddCustUtil.scala:40 []
|  ShuffledRDD[4] at sortByKey at rddCustUtil.scala:38 []
+-(1) MapPartitionsRDD[3] at map at rddCustUtil.scala:38 []
    |  MapPartitionsRDD[2] at map at rddCustUtil.scala:38 []
    |  /Data/ MapPartitionsRDD[1] at textFile at rddCustUtil.scala:35 []
    |  /Data/ HadoopRDD[0] at textFile at rddCustUtil.scala:35 []
|  MapPartitionsRDD[17] at map at rddCustUtil.scala:68 []
|  MapPartitionsRDD[16] at map at rddCustUtil.scala:68 []
|  MapPartitionsRDD[5] at map at rddCustUtil.scala:40 []
|  ShuffledRDD[4] at sortByKey at rddCustUtil.scala:38 []
+-(1) MapPartitionsRDD[3] at map at rddCustUtil.scala:38 []
    |  MapPartitionsRDD[2] at map at rddCustUtil.scala:38 []
    |  /Data/ MapPartitionsRDD[1] at textFile at rddCustUtil.scala:35 []
    |  /Data/ HadoopRDD[0] at textFile at rddCustUtil.scala:35 []
|  MapPartitionsRDD[19] at map at rddCustUtil.scala:73 []
|  MapPartitionsRDD[18] at map at rddCustUtil.scala:73 []
|  MapPartitionsRDD[5] at map at rddCustUtil.scala:40 []
|  ShuffledRDD[4] at sortByKey at rddCustUtil.scala:38 []
+-(1) MapPartitionsRDD[3] at map at rddCustUtil.scala:38 []
    |  MapPartitionsRDD[2] at map at rddCustUtil.scala:38 []
    |  /Data/ MapPartitionsRDD[1] at textFile at rddCustUtil.scala:35 []
    |  /Data/ HadoopRDD[0] at textFile at rddCustUtil.scala:35 []
</pre>

val input1=rawinput.map（u.split（“\t”））.map（x=>（x（6.trim（），x））.sortByKey（）
val input2=input1.map（x=>x._2.mkString（“\t”））
val x0=input2.map（u.split（“\t”））.map（x=>（x（6），x（0））
valx1=input2.map（u.split（“\t”））.map（x=>（x（6），x（1））
valx2=input2.map（u.split（“\t”））.map（x=>（x（6），x（2））
VALx3=input2.map（u.split（“\t”））.map（x=>（x（6），x（3））
val x4=input2.map（u.split（“\t”））.map（x=>（x（6），x（4））
val x5=input2.map（u.split（“\t”））.map（x=>（x（6），x（5））
valx6=input2.map（u.split（“\t”））.map（x=>（x（6），x（6））
val x=x0接头x1接头x2接头x3接头x4接头x5接头x6
val input1 = rawinput.map(_.split("\t")).map(x=>(x(6).trim(),x)).sortByKey()
val input2 = input1.map(x=> x._2.mkString("\t")).cache()
// continue as before

**沿袭图：**
（7） UnionRDD[25]位于rddCustUtil.scala的union处：78[]
|UnionRDD[24]位于rddCustUtil.scala的union处：78[]
|UnionRDD[23]位于rddCustUtil.scala:78[]
|UnionRDD[22]位于rddCustUtil.scala的union处：78[]
|UnionRDD[21]位于rddCustUtil.scala的union处：78[]
|UnionRDD[20]位于rddCustUtil.scala:78[]
|MapPartitionsRDD[7]位于rddCustUtil.scala的映射：43[]
|MapPartitionsRDD[6]位于rddCustUtil.scala的映射：43[]
|MapPartitionsRDD[5]位于rddCustUtil.scala的映射：40[]
|Shuffledd[4]位于sortByKey的rddCustUtil.scala:38[]
+-（1） MapPartitionsRDD[3]位于rddCustUtil.scala的映射：38[]
|MapPartitionsRDD[2]位于rddCustUtil.scala的映射：38[]
|/Data/MapPartitionsRDD[1]位于rddCustUtil.scala的textFile:35[]
|/Data/HadoopRDD[0]位于rddCustUtil.scala的textFile:35[]
|MapPartitionsRDD[9]位于rddCustUtil.scala的映射：48[]
|MapPartitionsRDD[8]位于rddCustUtil.scala的映射：48[]
|MapPartitionsRDD[5]位于rddCustUtil.scala的映射：40[]
|Shuffledd[4]位于sortByKey的rddCustUtil.scala:38[]
+-（1） MapPartitionsRDD[3]位于rddCustUtil.scala的映射：38[]
|MapPartitionsRDD[2]位于rddCustUtil.scala的映射：38[]
|/Data/MapPartitionsRDD[1]位于rddCustUtil.scala的textFile:35[]
|/Data/HadoopRDD[0]位于rddCustUtil.scala的textFile:35[]
|MapPartitionsRDD[11]位于rddCustUtil.scala的映射：53[]
|MapPartitionsRDD[10]位于rddCustUtil.scala的映射：53[]
|MapPartitionsRDD[5]位于rddCustUtil.scala的映射：40[]
|Shuffledd[4]位于sortByKey的rddCustUtil.scala:38[]
+-（1） MapPartitionsRDD[3]位于rddCustUtil.scala的映射：38[]
|MapPartitionsRDD[2]位于rddCustUtil.scala的映射：38[]
|/Data/MapPartitionsRDD[1]位于rddCustUtil.scala的textFile:35[]
|/Data/HadoopRDD[0]位于rddCustUtil.scala的textFile:35[]
|MapPartitionsRDD[13]位于rddCustUtil.scala的映射：58[]
|MapPartitionsRDD[12]位于rddCustUtil.scala的映射：58[]
|MapPartitionsRDD[5]位于rddCustUtil.scala的映射：40[]
|Shuffledd[4]位于sortByKey的rddCustUtil.scala:38[]
+-（1） MapPartitionsRDD[3]位于rddCustUtil.scala的映射：38[]
|MapPartitionsRDD[2]位于rddCustUtil.scala的映射：38[]
|/Data/MapPartitionsRDD[1]位于rddCustUtil.scala的textFile:35[]
|/Data/HadoopRDD[0]位于rddCustUtil.scala的textFile:35[]
|MapPartitionsRDD[15]位于rddCustUtil.scala的映射：63[]
|MapPartitionsRDD[14]位于rddCustUtil.scala的映射：63[]
|MapPartitionsRDD[5]位于rddCustUtil.scala的映射：40[]
|Shuffledd[4]位于sortByKey的rddCustUtil.scala:38[]
+-（1） MapPartitionsRDD[3]位于rddCustUtil.scala的映射：38[]
|MapPartitionsRDD[2]位于rddCustUtil.scala的映射：38[]
|/Data/MapPartitionsRDD[1]位于rddCustUtil.scala的textFile:35[]
|/Data/HadoopRDD[0]位于rddCustUtil.scala的textFile:35[]
|MapPartitionsRDD[17]位于rddCustUtil.scala的映射：68[]
|MapPartitionsRDD[16]位于rddCustUtil.scala的映射：68[]
|MapPartitionsRDD[5]位于rddCustUtil.scala的映射：40[]
|Shuffledd[4]位于sortByKey的rddCustUtil.scala:38[]
+-（1） MapPartitionsRDD[3]位于rddCustUtil.scala的映射：38[]
|MapPartitionsRDD[2]位于rddCustUtil.scala的映射：38[]
|/Data/MapPartitionsRDD[1]位于rddCustUtil.scala的textFile:35[]
|/Data/HadoopRDD[0]位于rddCustUtil.scala的textFile:35[]
|MapPartitionsRDD[19]位于rddCustUtil.scala的映射：73[]
|MapPartitionsRDD[18]位于rddCustUtil.scala的地图上：73[]
|MapPartitionsRDD[5]位于rddCustUtil.scala的映射：40[]
|Shuffledd[4]位于sortByKey的rddCustUtil.scala:38[]
+-（1） MapPartitionsRDD[3]位于rddCustUtil.scala的映射：38[]
|MapPartitionsRDD[2]位于rddCustUtil.scala的映射：38[]
|/Data/MapPartitionsRDD[1]位于rddCustUtil.scala的textFile:35[]
|/Data/HadoopRDD[0]位于rddCustUtil.scala的textFile:35[]

• 你能解释一下，在显示7 ShuffledRd[4]时，将执行多少个洗牌阶段吗
• 你能给我详细解释一下下面的DAG流程吗
• 这个手术贵吗

将执行多少个洗牌阶段

事实上，对数据进行排序所需的洗牌会发生7次，因为Spark的计算是惰性的，并且是按需运行的，除非缓存，否则将为DAG中需要它的每个分支重新计算。要解决这一问题（并使计算速度更快），您可以缓存（或者更一般地说，持久化）

input2

在多次使用之前：

+-(1) MapPartitionsRDD[3] at map at rddCustUtil.scala:38 []
    |  MapPartitionsRDD[2] at map at rddCustUtil.scala:38 []
    |  /Data/ MapPartitionsRDD[1] at textFile at rddCustUtil.scala:35 []
    |  /Data/ HadoopRDD[0] at textFile at rddCustUtil.scala:35 []
|  MapPartitionsRDD[9] at map at rddCustUtil.scala:48 []
|  MapPartitionsRDD[8] at map at rddCustUtil.scala:48 []
|  MapPartitionsRDD[5] at map at rddCustUtil.scala:40 []
|  ShuffledRDD[4] at sortByKey at rddCustUtil.scala:38 []

你能给我详细解释下DAG流程吗

每个

x

RDD都是使用以下计算“单独”计算的：

val x = rawinput.map(_.split("\t"))
  .keyBy(_(6).trim()) // extract key
  .flatMap{ case (k, arr) => arr.take(7).zipWithIndex.map((k, _)) } // flatMap into (key, (value, index))
  .sortBy { case (k, (_, index)) => (index, k) } // sort by index first, key second
  .map    { case (k, (value, _)) => (k, value) } // remove index, it was just used for sorting

它显示了从textFile创建

rawinput

的计算，然后是排序和三个

map

操作

然后，您有6个联合操作来联合这7个RDD

这个手术贵吗

是的，看起来应该是这样。如上所述，缓存可以大大加快速度，但有一种更好的方法可以实现这一点，而无需将RDD拆分为多个单独的RDD：

这将执行单个洗牌操作，并且不需要持久化数据。DAG如下所示：

---

非常感谢Tzach，但若我使用缓存或持久化方法，那个么该如何处理呢

(4) MapPartitionsRDD[9] at map at Test.scala:75 []
 |  MapPartitionsRDD[8] at sortBy at Test.scala:74 []
 |  ShuffledRDD[7] at sortBy at Test.scala:74 []
 +-(4) MapPartitionsRDD[4] at sortBy at Test.scala:74 []
    |  MapPartitionsRDD[3] at flatMap at Test.scala:73 []
    |  MapPartitionsRDD[2] at keyBy at Test.scala:72 []
    |  MapPartitionsRDD[1] at map at Test.scala:71 []
    |  ParallelCollectionRDD[0] at parallelize at Test.scala:64 []