Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Spark读取CSV实时路径_Csv_Apache Spark_Relative Path - Fatal编程技术网

Spark读取CSV实时路径

Spark读取CSV实时路径,csv,apache-spark,relative-path,Csv,Apache Spark,Relative Path,如何使用相对路径将CSV读入spark? 到目前为止,使用绝对路径工作得很好(1.6.2,2.0.1),但我需要通过相对路径加载数据 试图像这样读取文件 val mynewDf = spark.read .option("header", "true") .option("inferSchema", "true") .option("charset", "UTF-8") .option("delimiter", ";") .csv("~/myProject

如何使用相对路径将CSV读入spark? 到目前为止,使用绝对路径工作得很好(1.6.2,2.0.1),但我需要通过相对路径加载数据

试图像这样读取文件

val mynewDf = spark.read
    .option("header", "true")
    .option("inferSchema", "true")
    .option("charset", "UTF-8")
    .option("delimiter", ";")
    .csv("~/myProject/someFolder/data.csv")
导致以下异常

Path does not exist: file:/Users/geoHeil/source/to/scala/sparkProject/~/myProject/someFolder/data.csv;

因此,并非总是火花才是罪魁祸首;)我决定使用普通scala获取相对文件路径:

var path = "~/myProject/data/data.csv"
path = path.replaceFirst("^~", System.getProperty("user.home"))