Scala 如何将JSON中的键转换为小写?
我有一个流式查询,它以JSON格式从Kafka中读取数据,在驼峰情况下有1000多个键Scala 如何将JSON中的键转换为小写?,scala,apache-spark,apache-spark-sql,spark-structured-streaming,Scala,Apache Spark,Apache Spark Sql,Spark Structured Streaming,我有一个流式查询,它以JSON格式从Kafka中读取数据,在驼峰情况下有1000多个键 scala> kafka_df.printSchema() root |-- jsonData: struct (nullable = true) | |-- header: struct (nullable = true) | | |-- batch_id: string (nullable = true) | | |-- entity: string (null
scala> kafka_df.printSchema()
root
|-- jsonData: struct (nullable = true)
| |-- header: struct (nullable = true)
| | |-- batch_id: string (nullable = true)
| | |-- entity: string (nullable = true)
| | |-- time: integer (nullable = true)
| | |-- key: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- message_type: string (nullable = true)
| |-- body: string (nullable = true)
如何递归地将键更改为小写并转换回数据帧,以便我可以使用写入流进行写入?尝试以下操作:
def columnsToLowercase(schema: StructType): StructType = {
def recurRename(schema: StructType): Seq[StructField] =
schema.fields.map {
case StructField(name, dtype: StructType, nullable, meta) =>
StructField(name.toLowerCase, StructType(recurRename(dtype)), nullable, meta)
case StructField(name, dtype, nullable, meta) =>
StructField(name.toLowerCase, dtype, nullable, meta)
}
StructType(recurRename(schema))
}
val newDF = sparkSession.createDataFrame(dataFrame.rdd, columnsToLowercase(dataFrame.schema))
试试这个:
def columnsToLowercase(schema: StructType): StructType = {
def recurRename(schema: StructType): Seq[StructField] =
schema.fields.map {
case StructField(name, dtype: StructType, nullable, meta) =>
StructField(name.toLowerCase, StructType(recurRename(dtype)), nullable, meta)
case StructField(name, dtype, nullable, meta) =>
StructField(name.toLowerCase, dtype, nullable, meta)
}
StructType(recurRename(schema))
}
val newDF = sparkSession.createDataFrame(dataFrame.rdd, columnsToLowercase(dataFrame.schema))
你在说什么钥匙?这是
标题中的键
数组还是架构中的所有字段?您能否以驼峰式的形式展示JSON格式的数据示例以及预期的输出?您在谈论什么键?这是标题中的键
数组还是架构中的所有字段?您能否以驼峰格式显示JSON格式的数据示例以及预期的输出?