Scala Spark读取json
我的代码如下所示Scala Spark读取json,json,scala,apache-spark,Json,Scala,Apache Spark,我的代码如下所示 val sparkConf = new SparkConf().setAppName("Json Test").setMaster("local[*]") val sc = new SparkContext(sparkConf) val sqlContext = new org.apache.spark.sql.SQLContext(sc) import sqlContext.implicits._ val path = "/path/log.json" val df
val sparkConf = new SparkConf().setAppName("Json Test").setMaster("local[*]")
val sc = new SparkContext(sparkConf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
val path = "/path/log.json"
val df = sqlContext.read.json(path)
df.show()
+---+--------------------+----+-------------+
|COL| DATA|IFAM| KTM|
+---+--------------------+----+-------------+
| 21|[[2,30,0,null], [...| EQR|1430006400000|
+---+--------------------+----+-------------+
+---+--------------------+----+-------------+
|COL| DATA|IFAM| KTM|
+---+--------------------+----+-------------+
| 21|[[2,30,0,null], [...| EQR|1430006400000|
+---+--------------------+----+-------------+
+---+--------------------+----+-------------+
|COL| DATA|IFAM| KTM|
+---+--------------------+----+-------------+
| 21|[[2,30,0,null], [...| EQR|1430006400000|
+---+--------------------+----+-------------+
+---+--------------------+----+-------------+
|COL| DATA|IFAM| KTM|
+---+--------------------+----+-------------+
| 21|[[2,30,0,null], [...| EQR|1430006400000|
+---+--------------------+----+-------------+
+---+--------------------+----+-------------+
|COL| DATA|IFAM| KTM|
+---+--------------------+----+-------------+
| 21|[[2,30,0,null], [...| EQR|1430006400000|
+---+--------------------+----+-------------+
+---+--------------------+----+-------------+
|COL| DATA|IFAM| KTM|
+---+--------------------+----+-------------+
| 21|[[2,30,0,null], [...| EQR|1430006400000|
+---+--------------------+----+-------------+
我非常确定您的路径不正确。请检查文件是否位于指定的路径。Json有效。路径应该正确。但提供的Json无效。请更正示例Json，然后重试。
+---+--------------------+----+-------------+
|COL| DATA|IFAM| KTM|
+---+--------------------+----+-------------+
| 21|[[2,30,0,null], [...| EQR|1430006400000|
+---+--------------------+----+-------------+
+---+--------------------+----+-------------+
|COL| DATA|IFAM| KTM|
+---+--------------------+----+-------------+
| 21|[[2,30,0,null], [...| EQR|1430006400000|
+---+--------------------+----+-------------+
+---+--------------------+----+-------------+
|COL| DATA|IFAM| KTM|
+---+--------------------+----+-------------+
| 21|[[2,30,0,null], [...| EQR|1430006400000|
+---+--------------------+----+-------------+
+---+--------------------+----+-------------+
|COL| DATA|IFAM| KTM|
+---+--------------------+----+-------------+
| 21|[[2,30,0,null], [...| EQR|1430006400000|
+---+--------------------+----+-------------+
+---+--------------------+----+-------------+
|COL| DATA|IFAM| KTM|
+---+--------------------+----+-------------+
| 21|[[2,30,0,null], [...| EQR|1430006400000|
+---+--------------------+----+-------------+
object Test {
def main(args: Array[String]) {
val sparkConf = new SparkConf().setAppName("Json Test").setMaster("local[*]")
val sc = new SparkContext(sparkConf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
val path = "/home/test/Desktop/test.json"
val df = sqlContext.read.json(path)
df.show()
}
}
你的json有效吗？你能分享你的pom文件或sbt文件和json文件的样本吗？是的，这是一个有效的json，我使用eclipse scala ide编译和运行scala版本2.1json是正确的。我已经测试了它，并用上面的代码和json形成了数据框架。：）我更新的json是正确的，我认为管理员在json部分编辑错误，错误或者似乎不是关于json格式，而是关于spark设置。答案很有用，因为我的json有问题，我没有意识到！数据集填充为null，但现在已正确填充。