Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Spark JavaRdd到数据帧转换代码停止,没有错误_Java_Apache Spark_Dataframe_Rdd - Fatal编程技术网

Spark JavaRdd到数据帧转换代码停止,没有错误

Spark JavaRdd到数据帧转换代码停止,没有错误,java,apache-spark,dataframe,rdd,Java,Apache Spark,Dataframe,Rdd,我试图将JavaRDD类转换为DataFrame,以便以后可以将其保存到parquet文件中,但代码执行在createDataFrame函数调用时停止,当我处理异常时,它给出null。这是我的课 public static class PageRanksCase implements Serializable{ private String node; private Double importance; public void setNode(String no

我试图将
JavaRDD
类转换为
DataFrame
,以便以后可以将其保存到
parquet
文件中,但代码执行在
createDataFrame
函数调用时停止,当我处理异常时,它给出null。这是我的课

public static class PageRanksCase implements Serializable{

    private String node;
    private Double importance; 


    public void setNode(String node) { this.node = node; }
    public void setImportance(Double importance) { this.importance = importance; }

    public String getNode(String node) { return this.node; }
    public Double getImportance(String node) { return this.importance; }
}
这是我试图运行的代码,用于将类转换为
DataFrame

try{
    JavaRDD<PageRanksCase> finalData = GetTopNNodes(pairedrdd);
    System.out.println("coming here");
    DataFrame finalFrame = Service.sqlCtx().createDataFrame(finalData,PageRanksCase.class);
    System.out.println("coming here too");
    finalFrame.write().parquet(rdfanalyzer.spark.Configuration.storage() + "sib200PageRank.parquet");
}
catch(Exception e)
{ 
  System.out.println(e.getMessage());  // gives null here.
}

有人知道这里发生了什么吗?

我使用上面提到的示例重新实现了代码

public static JavaRDD<PageRanksCase> GetTopNNodes(JavaPairRDD<String,Double> pairedrdd){

    return pairedrdd.map(new Function<Tuple2<String,Double>, PageRanksCase>() {

        @Override
        public PageRanksCase call(Tuple2<String, Double> line) throws Exception {
            PageRanksCase pgrank  = new PageRanksCase();
            pgrank.setImportance(line._2);
            pgrank.setNode(line._1());
            return pgrank;
        }
    });
}