Mongodb 如何最好地处理将MongoRDD转换为DataFrame的模式冲突？

我试图从mongo数据库中读取一些文档，并在spark数据框架中解析模式。到目前为止，我已经成功地从mongo读取数据，并使用case类定义的模式将生成的mongoRDD转换为数据帧，但有一种情况是mongo集合有一个包含多个数据类型的字段（字符串数组和嵌套对象数组）。到目前为止，我只是将字段解析为字符串，然后使用spark sql的from_json（）解析新模式中的嵌套对象，但我发现，当字段不符合模式时，它会为模式中的所有字段返回null，而不仅仅是不符合的字段。有没有一种方法可以解析它，这样只有与模式不匹配的字段才会返回null

//creating mongo test data in mongo shell
db.createCollection("testColl")
db.testColl.insertMany([
    { "foo" : ["fooString1", "fooString2"], "bar" : "barString"},
    { "foo" : [{"uid" : "fooString1"}, {"uid" : "fooString2"}], "bar" : "barString"}
])


import com.mongodb.spark.config.ReadConfig
import org.apache.spark.sql.{DataFrame, Row}
import org.apache.spark.sql.catalyst.ScalaReflection
import org.apache.spark.sql.functions._
import com.mongodb.spark.MongoSpark
import org.apache.spark.sql.types.{StringType, StructField, StructType}

//mongo connector and read config
val testConfig = ReadConfig(Map("uri" -> "mongodb://some.mongo.db",
    "database" -> "testDB",
    "collection" -> "testColl"
  ))



//Option 1: 'lowest common denominator' case class - works, but leaves the nested struct type value as json that then needs additional parsing

case class stringArray (foo: Option[Seq[String]], bar: Option[String])
val df1 : DataFrame = MongoSpark.load(spark.sparkContext, testConfig).toDF[stringArray]
df1.show()
+--------------------+---------+
|                 foo|      bar|
+--------------------+---------+
|[fooString1, fooS...|barString|
|[{ "uid" : "fooSt...|barString|
+--------------------+---------+


//Option 2: accurate case class - fails with:
//com.mongodb.spark.exceptions.MongoTypeConversionException: Cannot cast STRING into a StructType(StructField(uid,StringType,true)) (value: BsonString{value='fooString1'})

case class fooDoc (uid: Option[String])
case class docArray (foo: Option[Seq[fooDoc]], bar: Option[String])
val df2 : DataFrame = MongoSpark.load(spark.sparkContext, testConfig).toDF[docArray]


//Option 3: map all rows to json string, then use from_json - why does return null for 'bar' in the case of the schema that doesn't fit?

val mrdd = MongoSpark.load(spark.sparkContext, testConfig)
val jsonRDD = mrdd.map(x => Row(x.toJson()))
val simpleSchema = StructType(Seq(StructField("wholeRecordJson", StringType, true)))
val schema = ScalaReflection.schemaFor[docArray].dataType.asInstanceOf[StructType]
val jsonDF = spark.createDataFrame(jsonRDD, simpleSchema)
val df3 = jsonDF.withColumn("parsed",from_json($"wholeRecordJson", schema))
df3.select("parsed.foo", "parsed.bar").show()
+--------------------+---------+
|                 foo|      bar|
+--------------------+---------+
|                null|     null|
|[[fooString1], [f...|barString|
+--------------------+---------+


//Desired results:
//desired outcome is for only the field not matching the schema (string type of 'foo') is null, but matching columns are populated

+--------------------+---------+
|                 foo|      bar|
+--------------------+---------+
|                null|barString|
|[[fooString1], [f...|barString|
+--------------------+---------+

---

不，要做到这一点并不容易，因为在同一文档集合中合并不兼容的模式是一种反模式，即使在Mongo中也是如此

解决这一问题的主要方法有三种：

修复MongoDB中的数据

发出一个“规范化”Mongo模式的查询，例如，删除不兼容类型的字段，或转换或重命名这些字段，等等

针对特定架构类型的文档向Mongo发出单独的查询。（Mongo具有可以根据字段类型进行过滤的查询运算符。）然后在Spark中进行后期处理，最后将数据合并到单个Spark数据集中