使用SparkR 1.5从RStudio中的hdfs读取大型文件(纯文本、xml、json、csv)的选项
我是Spark新手,我想知道除了下面的选项之外,是否还有其他选项可以使用SparkR从RStudio读取hdfs中存储的数据,或者我是否正确使用了它们。数据可以是任何类型(纯文本、csv、json、xml或任何包含关系表的数据库)和任何大小(1kb-几gb) 我知道textFile(sc,path)不应该再被使用了,但是除了read.df函数之外,还有其他可能读取此类数据吗 以下代码使用read.df和jsonFile,但jsonFile会产生错误:使用SparkR 1.5从RStudio中的hdfs读取大型文件(纯文本、xml、json、csv)的选项,r,sparkr,apache-spark-1.5,R,Sparkr,Apache Spark 1.5,我是Spark新手,我想知道除了下面的选项之外,是否还有其他选项可以使用SparkR从RStudio读取hdfs中存储的数据,或者我是否正确使用了它们。数据可以是任何类型(纯文本、csv、json、xml或任何包含关系表的数据库)和任何大小(1kb-几gb) 我知道textFile(sc,path)不应该再被使用了,但是除了read.df函数之外,还有其他可能读取此类数据吗 以下代码使用read.df和jsonFile,但jsonFile会产生错误: Sys.setenv(SPARK_HOME
Sys.setenv(SPARK_HOME = "C:\\Users\\--\\Downloads\\spark-1.5.0-bin-hadoop2.6")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
#load the Sparkr library
library(SparkR)
# Create a spark context and a SQL context
sc <- sparkR.init(master="local", sparkPackages="com.databricks:spark-csv_2.11:1.0.3")
sqlContext <- sparkRSQL.init(sc)
#create a sparkR DataFrame
df <- read.df(sqlContext, "hdfs://0.0.0.0:19000/people.json", source = "json")
df <- jsonFile(sqlContext, "hdfs://0.0.0.0:19000/people.json")
Sys.setenv(SPARK\u HOME=“C:\\Users\\\--\\Downloads\\SPARK-1.5.0-bin-hadoop2.6”)
.libPaths(c(file.path(Sys.getenv(“SPARK_HOME”)、“R”、“lib”)、.libPaths())
#加载Sparkr库
图书馆(SparkR)
#创建spark上下文和SQL上下文
sc试试看
%hadoop fs-put people.json/
%斯巴克
>人头(人)
您可能需要一个库来解析其他文件,如DataBricks CSV库:
然后,您将在加载包的情况下启动R,例如:
$sparkR--packages com.databricks:spark-csv_2.10:1.0.3
并按如下方式加载您的文件:
>df他在spark init中使用了这一点,他说的是从RStudio启动,而不是从命令行启动。
> df <- read.df(sqlContext, "hdfs://0.0.0.0:19000/README.txt", "text")
Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
java.lang.ClassNotFoundException: Failed to load class for data source: text.
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:67)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:87)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114)
at org.apache.spark.sql.api.r.SQLUtils$.loadDF(SQLUtils.scala:156)
at org.apache.spark.sql.api.r.SQLUtils.loadDF(SQLUtils.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:132)
at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:79)
at org.apache.spark.ap
> df <- jsonFile(sqlContext, "hdfs://0.0.0.0:19000/people.json")
Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :
java.io.IOException: No input paths specified in job
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:201)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:207)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfu
data <- SparkR:::textFile(sc, "hdfs://0.0.0.0:19000/people.json")
Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :
java.lang.IllegalArgumentException: java.net.URISyntaxException: Expected scheme-specific part at index 5: hdfs:
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.<init>(Path.java:172)
at org.apache.hadoop.fs.Path.<init>(Path.java:94)
at org.apache.hadoop.fs.Globber.glob(Globber.java:211)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1644)
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:257)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:207)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at or
% hadoop fs -put people.json /
% sparkR
> people <- read.df(sqlContext, "/people.json", "json")
> head(people)