Scala 在Spark rdd中格式化映射中的嵌套映射

我有一个如下所示的文本文件：

1007|CNSMR_CARD|1|1|1|1|1|1|1
1007|CNSMR_LOCL_IM_CHKG|1|1|1|1|1|1|1
1009|CNSMR_DIRCT_CHKG|4|4|4|4|4|1|1
1009|CNSMR_DIRCT_OTHR|4|4|4|4|4|1|1
1009|CNSMR_DIRCT_SAVG|4|4|4|4|4|1|1
1009|CNSMR_LOCL_IM_CHKG|4|4|4|4|4|1|1
1010|CNSMR_LOCL_IM_CHKG|1|1|1|1|1|1|1
1012|CNSMR_LOCL_IM_CHKG|1|1|1|1|1|1|1
1033|CNSMR_DIRCT_CHKG|1|1|1|1|2|1|1

val custFile = sc.textFile("custInfo.txt").map(line => line.split('|'))

val custPrd = custFile.map(a => (a(0), ((a(1)), Map("PRVCY_MAIL: " -> a(2), "PRVCY_CALL: " -> a(3), "PRVCY_SWP: " -> a(4), "PRVCY_FCRA: " -> a(5), "PRVCY_GLBA: " -> a(6), "PRVCY_PIPE: " -> a(7), "PRVCY_AFIL: " -> a(8)))))

val custGrp = custPrd.groupByKey

val custPrdGrp = custGrp.map{case (k, vals) => {val valsString = vals.mkString(", "); s"'$k' | {$valsString}" }}

'106' | {'CNSMR_LOCL_IM_CHKG': {PRVCY_MAIL: 4, PRVCY_GLBA: 4, PRVCY_FCRA: 4, PRVCY_AFIL: 1, PRVCY_PIPE: 1, PRVCY_CALL: 4, PRVCY_SWP: 4}}
'107' | {'CNSMR_DIRCT_CHKG': {PRVCY_MAIL: 1, PRVCY_GLBA: 1, PRVCY_FCRA: 1, PRVCY_AFIL: 1, PRVCY_PIPE: 1, PRVCY_CALL: 4, PRVCY_SWP: 1}}, {'CNSMR_DIRCT_SAVG': {PRVCY_MAIL: 1, PRVCY_GLBA: 1, PRVCY_FCRA: 1, PRVCY_AFIL: 1, PRVCY_PIPE: 1, PRVCY_CALL: 4, PRVCY_SWP: 1}}

然后创建一个rdd，如下所示：

1007|CNSMR_CARD|1|1|1|1|1|1|1
1007|CNSMR_LOCL_IM_CHKG|1|1|1|1|1|1|1
1009|CNSMR_DIRCT_CHKG|4|4|4|4|4|1|1
1009|CNSMR_DIRCT_OTHR|4|4|4|4|4|1|1
1009|CNSMR_DIRCT_SAVG|4|4|4|4|4|1|1
1009|CNSMR_LOCL_IM_CHKG|4|4|4|4|4|1|1
1010|CNSMR_LOCL_IM_CHKG|1|1|1|1|1|1|1
1012|CNSMR_LOCL_IM_CHKG|1|1|1|1|1|1|1
1033|CNSMR_DIRCT_CHKG|1|1|1|1|2|1|1

val custFile = sc.textFile("custInfo.txt").map(line => line.split('|'))

val custPrd = custFile.map(a => (a(0), ((a(1)), Map("PRVCY_MAIL: " -> a(2), "PRVCY_CALL: " -> a(3), "PRVCY_SWP: " -> a(4), "PRVCY_FCRA: " -> a(5), "PRVCY_GLBA: " -> a(6), "PRVCY_PIPE: " -> a(7), "PRVCY_AFIL: " -> a(8)))))

val custGrp = custPrd.groupByKey

val custPrdGrp = custGrp.map{case (k, vals) => {val valsString = vals.mkString(", "); s"'$k' | {$valsString}" }}

'106' | {'CNSMR_LOCL_IM_CHKG': {PRVCY_MAIL: 4, PRVCY_GLBA: 4, PRVCY_FCRA: 4, PRVCY_AFIL: 1, PRVCY_PIPE: 1, PRVCY_CALL: 4, PRVCY_SWP: 4}}
'107' | {'CNSMR_DIRCT_CHKG': {PRVCY_MAIL: 1, PRVCY_GLBA: 1, PRVCY_FCRA: 1, PRVCY_AFIL: 1, PRVCY_PIPE: 1, PRVCY_CALL: 4, PRVCY_SWP: 1}}, {'CNSMR_DIRCT_SAVG': {PRVCY_MAIL: 1, PRVCY_GLBA: 1, PRVCY_FCRA: 1, PRVCY_AFIL: 1, PRVCY_PIPE: 1, PRVCY_CALL: 4, PRVCY_SWP: 1}}

这给了我这个结果：

res4: Array[String] = Array(
'106' | {(CNSMR_LOCL_IM_CHKG,Map(PRVCY_MAIL:  -> 4, PRVCY_GLBA:  -> 4, PRVCY_FCRA:  -> 4, PRVCY_AFIL:  -> 1, PRVCY_PIPE:  -> 1, PRVCY_CALL:  -> 4, PRVCY_SWP:  -> 4))}, 
'107' | {(CNSMR_DIRCT_CHKG,Map(PRVCY_MAIL:  -> 1, PRVCY_GLBA:  -> 1, PRVCY_FCRA:  -> 1, PRVCY_AFIL:  -> 1, PRVCY_PIPE:  -> 1, PRVCY_CALL:  -> 4, PRVCY_SWP:  -> 1)), (CNSMR_DIRCT_SAVG,Map(PRVCY_MAIL:  -> 1, PRVCY_GLBA:  -> 1, PRVCY_FCRA:  -> 1, PRVCY_AFIL:  -> 1, PRVCY_PIPE:  -> 1, PRVCY_CALL:  -> 4, PRVCY_SWP:  -> 1))}

但我想要的是这样一个数组：

1007|CNSMR_CARD|1|1|1|1|1|1|1
1007|CNSMR_LOCL_IM_CHKG|1|1|1|1|1|1|1
1009|CNSMR_DIRCT_CHKG|4|4|4|4|4|1|1
1009|CNSMR_DIRCT_OTHR|4|4|4|4|4|1|1
1009|CNSMR_DIRCT_SAVG|4|4|4|4|4|1|1
1009|CNSMR_LOCL_IM_CHKG|4|4|4|4|4|1|1
1010|CNSMR_LOCL_IM_CHKG|1|1|1|1|1|1|1
1012|CNSMR_LOCL_IM_CHKG|1|1|1|1|1|1|1
1033|CNSMR_DIRCT_CHKG|1|1|1|1|2|1|1

val custFile = sc.textFile("custInfo.txt").map(line => line.split('|'))

val custPrd = custFile.map(a => (a(0), ((a(1)), Map("PRVCY_MAIL: " -> a(2), "PRVCY_CALL: " -> a(3), "PRVCY_SWP: " -> a(4), "PRVCY_FCRA: " -> a(5), "PRVCY_GLBA: " -> a(6), "PRVCY_PIPE: " -> a(7), "PRVCY_AFIL: " -> a(8)))))

val custGrp = custPrd.groupByKey

val custPrdGrp = custGrp.map{case (k, vals) => {val valsString = vals.mkString(", "); s"'$k' | {$valsString}" }}

'106' | {'CNSMR_LOCL_IM_CHKG': {PRVCY_MAIL: 4, PRVCY_GLBA: 4, PRVCY_FCRA: 4, PRVCY_AFIL: 1, PRVCY_PIPE: 1, PRVCY_CALL: 4, PRVCY_SWP: 4}}
'107' | {'CNSMR_DIRCT_CHKG': {PRVCY_MAIL: 1, PRVCY_GLBA: 1, PRVCY_FCRA: 1, PRVCY_AFIL: 1, PRVCY_PIPE: 1, PRVCY_CALL: 4, PRVCY_SWP: 1}}, {'CNSMR_DIRCT_SAVG': {PRVCY_MAIL: 1, PRVCY_GLBA: 1, PRVCY_FCRA: 1, PRVCY_AFIL: 1, PRVCY_PIPE: 1, PRVCY_CALL: 4, PRVCY_SWP: 1}}

要格式化第二个贴图，我尝试了类似的操作，但出现了一个错误：

    val custPrdGrp = custGrp.map{case (k, vals) => {val valsString = vals map { case (val1, val2, val3, val4, val5, val6, val7) => {val sets = vals.mkString(", "); s"$val1, $val2, $val3, $val4, $val5, $val6, $val7"}}.mkString(", "); s"'$k' | {$valsString}" }}

<console>:27: error: missing parameter type for expanded function
The argument types of an anonymous function must be fully known. (SLS 8.5)
Expected type was: ?
       val custPrdGrp = custGrp.map{case (k, vals) => {val valsString = vals map { case (val1, val2, val3, val4, val5, val6, val7) => {val sets = vals.mkString(", "); s"$val1, $val2, $val3, $val4, $val5, $val6, $val7"}}.mkString(", "); s"'$k' | {$valsString}" }}
                                                                                 ^

val-custPrdGrp=custGrp.map{case（k，vals）=>{val-valstring=vals-map{case（val1，val2，val3，val4，val5，val6，val7）=>{val-set=vals.mkString（“，”）；s“$val1，$val2，$val3，$val4，$val5，$val6，$val7”}.mkString（“，”；s“$k'{$valstring}”
：27:错误：缺少扩展函数的参数类型
匿名函数的参数类型必须完全已知。（补充说明8.5）
所需类型为：？
val custPrdGrp=custGrp.map{case（k，val）=>{val VALSTRING=val map{case（val1，val2，val3，val4，val5，val6，val7）=>{val sets=val.mkString（“，”；s“$val1，$val2，$val3，$val4，$val5，$val6，$val7”}}.mkString（“，”；s“$k'{$VALSTRING}
^

---

如何在Spark中格式化映射中的嵌套映射？

让我们从简单的

map[String，String]

val m: Map[String,String] = Map(
   "PRVCY_MAIL" -> "1", "PRVCY_GLBA" -> "1",
   "PRVCY_FCRA" -> "1", "PRVCY_AFIL" -> "1",
   "PRVCY_PIPE" -> "1", "PRVCY_CALL" -> "1",
   "PRVCY_SWP" -> "1"
)

请注意，我删除了格式元素，如

：

和whitscapes。在我看来，不需要买更干净的

现在我们可以定义两个小辅助对象：

def formatMap(sep: String = ": ",
    left: String = "{", right: String = "}")(m: Map[String, String]) = {
  val items = m.toSeq.map{case (k, v) => s"$k$sep$v"}.mkString(", ")
  s"$left$items$right"
}

让我们检查一下它是如何工作的

scala> formatMap()(m)
res50: String = {PRVCY_CALL: 1, PRVCY_SWP: 1, PRVCY_MAIL: 1, PRVCY_AFIL: 1, PRVCY_FCRA: 1, PRVCY_PIPE: 1, PRVCY_GLBA: 1}

scala> formatMap(sep="=")(m)
res51: String = {PRVCY_CALL=1, PRVCY_SWP=1, PRVCY_MAIL=1, PRVCY_AFIL=1, PRVCY_FCRA=1, PRVCY_PIPE=1, PRVCY_GLBA=1}

scala> formatMap(sep="|", left="[", right="]")(m)
res52: String = [PRVCY_CALL|1, PRVCY_SWP|1, PRVCY_MAIL|1, PRVCY_AFIL|1, PRVCY_FCRA|1, PRVCY_PIPE|1, PRVCY_GLBA|1]

现在让我们清理一下你已经有的东西。首先，让我们提取名称：

val keys = Array(
   "PRVCY_MAIL", "PRVCY_CALL", "PRVCY_SWP", "PRVCY_FCRA",
   "PRVCY_GLBA", "PRVCY_PIPE", "PRVCY_AFIL"
)

重写映射：

val custPrd = custFile.map(a => (a(0), (a(1), keys.zip(a.drop(2)).toMap)))

像以前一样分组

val custGrp = custPrd.groupByKey

和地图

val custPrdGrp = custGrp.map{case (k, vals) => {
  val valsString = vals.map{case (id, m) => {
    val fmtM = formatMap()(m)
    s"'$id': $fmtM"
  }}.mkString(", ")
  s"'$k' | {$valsString}"
}}

快速检查：

scala> custPrdGrp.first
res56: String = '1012' | {'CNSMR_LOCL_IM_CHKG': {PRVCY_CALL: 1, PRVCY_SWP: 1, PRVCY_MAIL: 1, PRVCY_AFIL: 1, PRVCY_FCRA: 1, PRVCY_PIPE: 1, PRVCY_GLBA: 1}}

您可能应该以类似于我为

formatMap

所做的方式提取上面使用的匿名函数，但我将把它作为练习留给您