Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/scala/17.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Java 如何为Spark中的SaveAsSequenceFile方法提供编解码器？_Java_Scala_Hadoop_Apache Spark - Fatal编程技术网
Fatal编程技术网
  • java/
  • Java 如何为Spark中的SaveAsSequenceFile方法提供编解码器？

Java 如何为Spark中的SaveAsSequenceFile方法提供编解码器？

java scala hadoop apache-spark

Java 如何为Spark中的SaveAsSequenceFile方法提供编解码器？,java,scala,hadoop,apache-spark,Java,Scala,Hadoop,Apache Spark,我试图弄清楚如何在ApacheSpark中将编解码器传递给saveAsSequenceFile方法。下面是我试图运行的代码。我正在运行Scala 2.10.4、Spark 1.0.0、Java 1.7.60和Apache Hadoop 2.4.0 val rdd:RDD[(String, String)] = sc.sequenceFile(secPath, classOf[Text],

我试图弄清楚如何在ApacheSpark中将编解码器传递给saveAsSequenceFile方法。下面是我试图运行的代码。我正在运行Scala 2.10.4、Spark 1.0.0、Java 1.7.60和Apache Hadoop 2.4.0

val rdd:RDD[(String, String)] = sc.sequenceFile(secPath,
                                classOf[Text],
                                classOf[Text]
                               ).map { case (k,v) => (k.toString, v.toString)}

val sortedOutput = rdd.sortByKey(true, 1)

sortedOutput.saveAsSequenceFile(secPathOut)
我的问题是我不熟悉Spark和Scala。我不明白javadoc对于传递给saveAsSequenceFile方法的codec变量意味着什么

def saveAsSequenceFile(path: String, codec: Option[Class[_ <: CompressionCodec]] = None): Unit 

def saveAsSequenceFile（路径：字符串，编解码器：选项[Class[\up>The


sc.parallelize(List((1,2))).saveAsSequenceFile("path",Some(classOf[GzipCodec]))