Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/scala/19.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181

Warning: file_get_contents(/data/phpspider/zhask/data//catemap/0/svn/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Scala 将RDD解析为键值对_Scala_Apache Spark_Apache Spark Sql - Fatal编程技术网

Scala 将RDD解析为键值对

Scala 将RDD解析为键值对,scala,apache-spark,apache-spark-sql,Scala,Apache Spark,Apache Spark Sql,我的csv文件中有如下数据 图例1、a1、10、a2、20、a3、30 键2,b1,50,b2,60 创建一个RDD并转换为另一个RDD,其中o/p应如下所示 键-1,a1 键-1,a2 钥匙-1,a3 键2,b1 键2,b2请根据上述示例查找以下代码 val rdd = sc.textFile("path to the csv file") // here sc is sparkContext val rdd1 = rdd.mapPartitions(itr => { itr

我的csv文件中有如下数据

图例1、a1、10、a2、20、a3、30

键2,b1,50,b2,60

创建一个RDD并转换为另一个RDD,其中o/p应如下所示

键-1,a1

键-1,a2

钥匙-1,a3

键2,b1


键2,b2

请根据上述示例查找以下代码

val rdd = sc.textFile("path to the csv file") // here sc is sparkContext
val rdd1 = rdd.mapPartitions(itr => {
      itr.map(_.split(",")).map(_.toList).map(list => (list.head, list.tail))
 })
val rdd2 = rdd1.mapValues(tail => tail.zipWithIndex).map(tuple => (tuple._1, 
       tuple._2.filter(x => x._2 % 2 ==0).map(_._1)))
val rdd3 = rdd2.mapPartitions(itr => {
      itr.flatMap(xx => xx._2.map(k => (xx._1, k)))
})
rdd3.foreach(println)
(Key-2, b1)
(Key-2, b2)
(Key-1, a1)
(Key-1, a2)
(Key-1, a3)