Apache spark 如何在spark中组合两列数据集

我有这样一个spark数据集：

> df.show()
+------+------+
| No1  | No2  |
+------+------+
| 001  | null |
| 002  | 002  |
| 003  | 004  |
| null | 005  |
+------+------+

我想得到一个新的列No3，它包含列No1和No2中的值，条件是复制No1，如果它有一个值，否则如果它为null，则使用No2中的值

---

如何执行此操作？

您可以检查

No1

列是否为

null

。如果其

null

则从

No2

import org.apache.spark.sql.functions._

val data = spark.sparkContext.parallelize(Seq(
  ("001", null),
  ("002", "002"),
  ("003", "004"),
  (null, "005")
)).toDF("No1", "No2")

val resultDf = data.withColumn("No3", when($"No1".isNull, $"No2").otherwise($"No1"))

resultDf.show

输出：

+----+----+---+
|No1 |No2 |No3|
+----+----+---+
|001 |null|001|
|002 |002 |002|
|003 |004 |003|
|null|005 |005|
+----+----+---+

---

希望这有帮助

我想你要找的是联合

import org.apache.spark.sql.functions._

val data = spark.sparkContext.parallelize(Seq(
  ("001", null),
  ("002", "002"),
  ("003", "004"),
  (null, "005")
)).toDF("No1", "No2")

val resultDf = data.withColumn("No3", coalesce($"No1", $"No2"))

resultDf.show

import org.apache.spark.sql.functions._

val data = spark.sparkContext.parallelize(Seq(
  ("001", null),
  ("002", "002"),
  ("003", "004"),
  (null, "005")
)).toDF("No1", "No2")

val resultDf = data.withColumn("No3", coalesce($"No1", $"No2"))

resultDf.show