使用spark sql将json数据加载到配置单元
我无法将json数据推入配置单元,下面是示例json数据和我的工作。请给我推荐丢失的那个 json数据使用spark sql将json数据加载到配置单元,json,scala,apache-spark,hive,apache-spark-sql,Json,Scala,Apache Spark,Hive,Apache Spark Sql,我无法将json数据推入配置单元,下面是示例json数据和我的工作。请给我推荐丢失的那个 json数据 { "Employees" : [ { "userId":"rirani", "jobTitleName":"Developer", "firstName":"Romin", "lastName":"Irani", "preferredFullName":"Romin Irani", "employeeCode":"E1", "region":"CA", "phoneNumber":"
{
"Employees" : [
{
"userId":"rirani",
"jobTitleName":"Developer",
"firstName":"Romin",
"lastName":"Irani",
"preferredFullName":"Romin Irani",
"employeeCode":"E1",
"region":"CA",
"phoneNumber":"408-1234567",
"emailAddress":"romin.k.irani@gmail.com"
},
{
"userId":"nirani",
"jobTitleName":"Developer",
"firstName":"Neil",
"lastName":"Irani",
"preferredFullName":"Neil Irani",
"employeeCode":"E2",
"region":"CA",
"phoneNumber":"408-1111111",
"emailAddress":"neilrirani@gmail.com"
},
{
"userId":"thanks",
"jobTitleName":"Program Directory",
"firstName":"Tom",
"lastName":"Hanks",
"preferredFullName":"Tom Hanks",
"employeeCode":"E3",
"region":"CA",
"phoneNumber":"408-2222222",
"emailAddress":"tomhanks@gmail.com"
}
]
}
我尝试使用sqlcontext和jsonFile方法来加载,但未能解析json
val f = sqlc.jsonFile("file:///home/vm/Downloads/emp.json")
f.show
error is : java.lang.RuntimeException: Failed to parse a value for data type StructType() (current token: VALUE_STRING)
我尝试了不同的方法,能够破解并获得模式
val files = sc.wholeTextFiles("file:///home/vm/Downloads/emp.json")
val jsonData = files.map(x => x._2)
sqlc.jsonRDD(jsonData).registerTempTable("employee")
val emp= sqlc.sql("select Employees[1].userId as ID,Employees[1].jobTitleName as Title,Employees[1].firstName as FirstName,Employees[1].lastName as LastName,Employees[1].preferredFullName as PeferedName,Employees[1].employeeCode as empCode,Employees[1].region as Region,Employees[1].phoneNumber as Phone,Employees[1].emailAddress as email from employee")
emp.show // displays all the values
我能够为每个记录分别获取数据和模式,但我缺少一个获取所有数据并加载到hive的想法
任何帮助或建议都非常感谢。SparkSQL只支持在文件每行包含一个JSON对象时读取JSON文件 您的文件应该是这样的(严格来说,它不是一个正确的JSON文件) 请看一看突出的部分。不要认为这是优先考虑的事情,只是记录在案 你有两个选择
SQLContext.jsonFile
已被弃用,请使用SQLContext.read.json
SparkSQL仅支持在文件每行包含一个JSON对象时读取JSON文件 您的文件应该是这样的(严格来说,它不是一个正确的JSON文件) 请看一看突出的部分。不要认为这是优先考虑的事情,只是记录在案 你有两个选择
SQLContext.jsonFile
已被弃用,请使用SQLContext.read.json
这是一个破烂的答案
val files = sc.wholeTextFiles("file:///home/vm/Downloads/emp.json")
val jsonData = files.map(x => x._2)
import org.apache.spark.sql.hive._
import org.apache.spark.sql.hive.HiveContext
val hc=new HiveContext(sc)
hc.jsonRDD(jsonData).registerTempTable("employee")
val fuldf=hc.jsonRDD(jsonData)
val dfemp=fuldf.select(explode(col("Employees")))
dfemp.saveAsTable("empdummy")
val df=sql("select * from empdummy")
df.select ("_c0.userId","_c0.jobTitleName","_c0.firstName","_c0.lastName","_c0.preferredFullName","_c0.employeeCode","_c0.region","_c0.phoneNumber","_c0.emailAddress").saveAsTable("dummytab")
任何优化上述代码的建议。以下是破解的答案
val files = sc.wholeTextFiles("file:///home/vm/Downloads/emp.json")
val jsonData = files.map(x => x._2)
import org.apache.spark.sql.hive._
import org.apache.spark.sql.hive.HiveContext
val hc=new HiveContext(sc)
hc.jsonRDD(jsonData).registerTempTable("employee")
val fuldf=hc.jsonRDD(jsonData)
val dfemp=fuldf.select(explode(col("Employees")))
dfemp.saveAsTable("empdummy")
val df=sql("select * from empdummy")
df.select ("_c0.userId","_c0.jobTitleName","_c0.firstName","_c0.lastName","_c0.preferredFullName","_c0.employeeCode","_c0.region","_c0.phoneNumber","_c0.emailAddress").saveAsTable("dummytab")
任何优化上述代码的建议。感谢Pranav让我的概念更加清晰,但我对解决这个json需求感到困惑,请帮助我。手动将所有json转换为每行一个对象是比较困难的任务。感谢Pranav让我的概念更加清晰,但我对解决json需求感到困惑,请帮助我。手动将所有json转换为每行一个对象是一项更困难的任务。
val files = sc.wholeTextFiles("file:///home/vm/Downloads/emp.json")
val jsonData = files.map(x => x._2)
import org.apache.spark.sql.hive._
import org.apache.spark.sql.hive.HiveContext
val hc=new HiveContext(sc)
hc.jsonRDD(jsonData).registerTempTable("employee")
val fuldf=hc.jsonRDD(jsonData)
val dfemp=fuldf.select(explode(col("Employees")))
dfemp.saveAsTable("empdummy")
val df=sql("select * from empdummy")
df.select ("_c0.userId","_c0.jobTitleName","_c0.firstName","_c0.lastName","_c0.preferredFullName","_c0.employeeCode","_c0.region","_c0.phoneNumber","_c0.emailAddress").saveAsTable("dummytab")