Scala Spark Dataframe-如何操作具有嵌套结构的行？

我有这样一个spark数据框：

+---------+---------+---------------------------------------------------+
|which_one|  matchID|                                        information|  
+---------+---------+---------------------------------------------------+
|    First|      123|[[1.2, 4.5, 837], [1.4, 4.8, 123], [4.1, 4.7, 143]]|   
|    First|      234|[[4.8, 8.9, 234], [1.1, 4.2, 321], [3.9, 5.7, 521]]|   
|   Second|      345|[[7.7, 8.1, 457], [4.5, 4.9, 345], [1.9, 2.8, 776]]|
+---------+---------+---------------------------------------------------+

对于每个matchID，我需要找到相应的信息列表。然后，使用“哪一个”提取我需要的信息

例如，对于matchID=123，它对应于[1.4,4.8123]。那么，这意味着我需要第一个val 1.4

对于matchID=234，它对应于[4.8,8.9234]。那么，这意味着我需要第一个val 4.8

对于matchID=345，它对应于[4.5,4.9,345]。那么，这意味着我需要第二个val 4.9

我想要的结果表是在末尾添加一个新列：

+---------+---------+---------------------------------------------------+---+
|which_one|  matchID|                                        information|res|  
+---------+---------+---------------------------------------------------+---+
|    First|      123|[[1.2, 4.5, 837], [1.4, 4.8, 123], [4.1, 4.7, 143]]|1.4|   
|    First|      234|[[4.8, 8.9, 234], [1.1, 4.2, 321], [3.9, 5.7, 521]]|4.8|   
|   Second|      345|[[7.7, 8.1, 457], [4.5, 4.9, 345], [1.9, 2.8, 776]]|4.9|
+---------+---------+---------------------------------------------------+---+

我尝试使用foreach和rdd循环遍历每一行。然而，它似乎无能为力，因为我找不到在信息列中搜索匹配ID的方法。我在网上搜索了几个小时，还没有找到有用的资源。欢迎提供任何建议或提示。

函数“分解”可用于将列“信息”转换为数组类型，然后按“匹配ID”筛选新列：

输出：

+---------+-------+---------------------------------------------------------+-----------------+-----------+
|which_one|matchID|information                                              |exploded         |indexColumn|
+---------+-------+---------------------------------------------------------+-----------------+-----------+
|First    |123    |[[1.2, 4.5, 837.0], [1.4, 4.8, 123.0], [4.1, 4.7, 143.0]]|[1.2, 4.5, 837.0]|1          |
|First    |123    |[[1.2, 4.5, 837.0], [1.4, 4.8, 123.0], [4.1, 4.7, 143.0]]|[1.4, 4.8, 123.0]|1          |
|First    |123    |[[1.2, 4.5, 837.0], [1.4, 4.8, 123.0], [4.1, 4.7, 143.0]]|[4.1, 4.7, 143.0]|1          |
|First    |234    |[[4.8, 8.9, 234.0], [1.1, 4.2, 321.0], [3.9, 5.7, 521.0]]|[4.8, 8.9, 234.0]|1          |
|First    |234    |[[4.8, 8.9, 234.0], [1.1, 4.2, 321.0], [3.9, 5.7, 521.0]]|[1.1, 4.2, 321.0]|1          |
|First    |234    |[[4.8, 8.9, 234.0], [1.1, 4.2, 321.0], [3.9, 5.7, 521.0]]|[3.9, 5.7, 521.0]|1          |
|Second   |345    |[[7.7, 8.1, 457.0], [4.5, 4.9, 345.0], [1.9, 2.8, 776.0]]|[7.7, 8.1, 457.0]|2          |
|Second   |345    |[[7.7, 8.1, 457.0], [4.5, 4.9, 345.0], [1.9, 2.8, 776.0]]|[4.5, 4.9, 345.0]|2          |
|Second   |345    |[[7.7, 8.1, 457.0], [4.5, 4.9, 345.0], [1.9, 2.8, 776.0]]|[1.9, 2.8, 776.0]|2          |
+---------+-------+---------------------------------------------------------+-----------------+-----------+

+---------+-------+---------------------------------------------------------+---+
|which_one|matchID|information                                              |res|
+---------+-------+---------------------------------------------------------+---+
|First    |123    |[[1.2, 4.5, 837.0], [1.4, 4.8, 123.0], [4.1, 4.7, 143.0]]|1.4|
|First    |234    |[[4.8, 8.9, 234.0], [1.1, 4.2, 321.0], [3.9, 5.7, 521.0]]|4.8|
|Second   |345    |[[7.7, 8.1, 457.0], [4.5, 4.9, 345.0], [1.9, 2.8, 776.0]]|4.9|
+---------+-------+---------------------------------------------------------+---+

---

非常感谢。我以前不知道会爆炸。这肯定能解决问题。

+---------+-------+---------------------------------------------------------+-----------------+-----------+
|which_one|matchID|information                                              |exploded         |indexColumn|
+---------+-------+---------------------------------------------------------+-----------------+-----------+
|First    |123    |[[1.2, 4.5, 837.0], [1.4, 4.8, 123.0], [4.1, 4.7, 143.0]]|[1.2, 4.5, 837.0]|1          |
|First    |123    |[[1.2, 4.5, 837.0], [1.4, 4.8, 123.0], [4.1, 4.7, 143.0]]|[1.4, 4.8, 123.0]|1          |
|First    |123    |[[1.2, 4.5, 837.0], [1.4, 4.8, 123.0], [4.1, 4.7, 143.0]]|[4.1, 4.7, 143.0]|1          |
|First    |234    |[[4.8, 8.9, 234.0], [1.1, 4.2, 321.0], [3.9, 5.7, 521.0]]|[4.8, 8.9, 234.0]|1          |
|First    |234    |[[4.8, 8.9, 234.0], [1.1, 4.2, 321.0], [3.9, 5.7, 521.0]]|[1.1, 4.2, 321.0]|1          |
|First    |234    |[[4.8, 8.9, 234.0], [1.1, 4.2, 321.0], [3.9, 5.7, 521.0]]|[3.9, 5.7, 521.0]|1          |
|Second   |345    |[[7.7, 8.1, 457.0], [4.5, 4.9, 345.0], [1.9, 2.8, 776.0]]|[7.7, 8.1, 457.0]|2          |
|Second   |345    |[[7.7, 8.1, 457.0], [4.5, 4.9, 345.0], [1.9, 2.8, 776.0]]|[4.5, 4.9, 345.0]|2          |
|Second   |345    |[[7.7, 8.1, 457.0], [4.5, 4.9, 345.0], [1.9, 2.8, 776.0]]|[1.9, 2.8, 776.0]|2          |
+---------+-------+---------------------------------------------------------+-----------------+-----------+

+---------+-------+---------------------------------------------------------+---+
|which_one|matchID|information                                              |res|
+---------+-------+---------------------------------------------------------+---+
|First    |123    |[[1.2, 4.5, 837.0], [1.4, 4.8, 123.0], [4.1, 4.7, 143.0]]|1.4|
|First    |234    |[[4.8, 8.9, 234.0], [1.1, 4.2, 321.0], [3.9, 5.7, 521.0]]|4.8|
|Second   |345    |[[7.7, 8.1, 457.0], [4.5, 4.9, 345.0], [1.9, 2.8, 776.0]]|4.9|
+---------+-------+---------------------------------------------------------+---+