Json Scala-Spark-如何将包含一个字符串列的数据帧转换为包含rigth类型列的DF？

我现在面临一个我无法解决的问题。 我使用的是Spark 1.6

我有一个文本数据框，其中一列包含一个字符串JSON和许多字段。 根据我从正确的Json推断出的一些模式，一些字段必须推断为字符串，另一些字段必须推断为数组，还有一些字段必须推断为长字段：

 {"eventid":"3bc1c5d2-c10f-48d6-8b35-05db8665415c","email":"test@test.com","prices_vat":["20295930","20295930"]}

我只成功地将它转换为带有字符串列的字段的df。 我无法将它转换为正确的类型

所需架构位于df_架构中。 列“value”包含我需要解析的字符串JSON。 这是我的密码：

     var b = sqlContext.createDataFrame(df_txt.rdd,df_schema)
     val z= {
     b.select( b.columns.map(c => get_json_object(b("value"), s"$$.$c").alias(c)): _*)
     }
    var c = sqlContext.createDataFrame(z.rdd,df_schema)
    c.show(1)

我以这个异常结束，因为字段“prices_vat”中的数组被理解为字符串，而不是像df_模式那样的数组：

   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 38.0 failed 1 times, most recent failure: Lost task 0.0 in stage 38.0 (TID 32, localhost): scala.MatchError: ["20295930","20295930"] (of class java.lang.String)
at org.apache.spark.sql.catalyst.CatalystTypeConverters$ArrayConverter.toCatalystImpl(CatalystTypeConverters.scala:159)
at org.apache.spark.sql.catalyst.CatalystTypeConverters$ArrayConverter.toCatalystImpl(CatalystTypeConverters.scala:153)
at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102)
at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:260)
at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:250)
at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102)
at org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$2.apply(CatalystTypeConverters.scala:401)
at org.apache.spark.sql.SQLContext$$anonfun$6.apply(SQLContext.scala:492)
at org.apache.spark.sql.SQLContext$$anonfun$6.apply(SQLContext.scala:492)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

---

请帮帮我

幸运的是，Spark有一些处理JSON数据的内置功能：

scala> val jsonRDD = sc.parallelize(
     |      """{"eventid":"3bc1c5d2-c10f-48d6-8b35-05db8665415c","email":"test@test.com","prices_vat":["20295930","20295930"]}""" :: Nil)
jsonRDD: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[8] at parallelize at <console>:27

scala> val df = sqlContext.read.json(jsonRDD)
df: org.apache.spark.sql.DataFrame = [email: string, eventid: string, prices_vat: array<string>]

scala> df.show
+-------------+--------------------+--------------------+
|        email|             eventid|          prices_vat|
+-------------+--------------------+--------------------+
|test@test.com|3bc1c5d2-c10f-48d...|[20295930, 20295930]|
+-------------+--------------------+--------------------+


scala> df.printSchema
root
 |-- email: string (nullable = true)
 |-- eventid: string (nullable = true)
 |-- prices_vat: array (nullable = true)
 |    |-- element: string (containsNull = true)

如果您在

数据帧中已经有了json，那么您可以这样做：

scala> import org.apache.spark.sql.Row
import org.apache.spark.sql.Row

scala> val df = sc.parallelize(
     |      """{"eventid":"3bc1c5d2-c10f-48d6-8b35-05db8665415c","email":"test@test.com","prices_vat":[20295930,20295930]}""" :: Nil).toDF("json")
df: org.apache.spark.sql.DataFrame = [json: string]

scala> df.show
+--------------------+
|                json|
+--------------------+
|{"eventid":"3bc1c...|
+--------------------+


scala> val rdd = df.rdd.map{case Row(json: String) => json}
rdd: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[43] at map at <console>:30

scala> val outDF = sqlContext.read.json(rdd)
outDF: org.apache.spark.sql.DataFrame = [email: string, eventid: string, prices_vat: array<bigint>]

scala> outDF.show
+-------------+--------------------+--------------------+
|        email|             eventid|          prices_vat|
+-------------+--------------------+--------------------+
|test@test.com|3bc1c5d2-c10f-48d...|[20295930, 20295930]|
+-------------+--------------------+--------------------+

scala>import org.apache.spark.sql.Row
导入org.apache.spark.sql.Row
scala>val df=sc.parallelize(
|“{”事件ID:“3bc1c5d2-c10f-48d6-8b35-05db8665415c”，“电子邮件：”test@test.com“，“价格/增值税”：[2029593020295930]}”“：：Nil.toDF（“json”）
df:org.apache.spark.sql.DataFrame=[json:string]
scala>df.show
+--------------------+
|json|
+--------------------+
|{“eventid”：“3bc1c…”|
+--------------------+
scala>val-rdd=df.rdd.map{case行（json:String）=>json}
rdd:org.apache.spark.rdd.rdd[String]=MapPartitionsRDD[43]位于映射位置：30
scala>val outDF=sqlContext.read.json（rdd）
outDF:org.apache.spark.sql.DataFrame=[电子邮件：string，eventid:string，prices\u-vat:array]
scala>outDF.show
+-------------+--------------------+--------------------+
|电子邮件|事件ID |价格|增值税|
+-------------+--------------------+--------------------+
|test@test.com|3bc1c5d2-c10f-48d…|[20295930，20295930]|
+-------------+--------------------+--------------------+
多亏了evan058，我们找到了解决这个问题的方法。
将此添加到我的代码中似乎有效：

var y= df_txt.select("value").rdd.map(r => r(0).asInstanceOf[String]).collect()
var o = sc.parallelize(y)
val r = sqlContext.read.json(o)

使用RDD使用对象映射器进行映射是一个选项吗？无论您提供什么解决方案，我忘了说我不想重新读取JSON。我已经有一个文本DF，其中一列包含JSON。我需要将DF_txt转换为某个JSON DF。您是对的！将此添加到我的代码中似乎有效：var y=DF_txt.select（“value”）.rdd.map（r=>r（0）.asInstanceOf[String]）.collect（）var o=sc.parallelize（y）val r=sqlContext.read.json（o）是的，您只需要访问底层rdd。如果我的解决方案帮助您解决了问题，请投票/接受它。@wymeka我添加了在DF中使用json的代码
var y= df_txt.select("value").rdd.map(r => r(0).asInstanceOf[String]).collect()
var o = sc.parallelize(y)
val r = sqlContext.read.json(o)