Scala 获取'Any'而不是Seq[DataFrame]
我想稍微改进以下代码:Scala 获取'Any'而不是Seq[DataFrame],scala,apache-spark,apache-spark-sql,Scala,Apache Spark,Apache Spark Sql,我想稍微改进以下代码: val filePathsList = Seq("path_1","path_2) var seqdf = filePathsList.map(path => { try { sqlContext.read.format("json").load(path) } catch { case e: Exception => e.printStackTrace() } })
val filePathsList = Seq("path_1","path_2)
var seqdf = filePathsList.map(path => {
try {
sqlContext.read.format("json").load(path)
} catch {
case e: Exception => e.printStackTrace()
}
})
现在seqdf的类型为Any,因为我添加了try。。。接住我需要seqdf是Seq[DataFrame]类型,同时维护try。。。捕获路径上不存在json文件的情况。如何获得它?使用Scala Try和getOrElse,这比Java Try catch更有效。如果找不到路径,则可以返回空数据帧
import sqlContext.implicits._
var seqdf = filePathsList.map(path => {
Try(
sqlContext.read.format("json").load(path)
) getOrElse Seq.empty[String].toDF()
})
希望这有帮助使用平面图和选项自动清除无效值
var seqdf = filePathsList.flatMap(path => {
try {
Some(sqlContext.read.format("json").load(path))
} catch { case e: Exception =>
e.printStackTrace()
None
}
})
如果您并不真正关心堆栈跟踪,那么下面将显示一个更简洁的版本
var seqdf = filePathsList.flatMap(path => {
Try(sqlContext.read.format("json").load(path)).toOption
})