Apache spark 如何在spark代码中用Java8编写mapFunction？错误：映射（函数1，编码器）不适用于参数

我有一个映射函数，如下所示

Function1<CompanyInfoRecordNew, CompanyInfoTransformedRecord> mapFunction = ( record ) ->{

        CompanyInfoTransformedRecord  transRec = new CompanyInfoTransformedRecord();
        //popluate 
    return transRec ;
    };

---

由于错误而无法进行显示，如何使显示正常工作

您的导入错误，而不是：

import org.apache.calcite.linq4j.function.Function1;

用这个

org.apache.spark.api.java.function.MapFunction;

baseDs.map(new MapFunction<CompanyInfoRecordNew, CompanyInfoTransformedRecord>() {...}, encoder);

---

要获取一个对象并返回对象列表，我们需要使用flatmap而不是map，如下所示

FlatMapFunction<Original, Derived>  flatMapFunction = ( record ) ->{

            List<Derived>  transRecList = new ArrayList<Derived>();

            Derived  transRec = new Derived();

            transRecList.add(transRec);

            Derived  transRec2 = new Derived();
                transRecList.add(transRec2);

            return (Iterator<Derived>) transRecList.iterator();
        };

---

数据集Deriveds=baseDs.flatMapflatMapFunction，comanyInfoTransEncoder

FunctionX是Scala特有的，MapFunction是Java特有的。org.apache.spark.api.Java.function.function用于JavaRDD。Dataset采用scala.Function[1-22]或org.apache.spark.api.java.Function。MapFunction您能创建一个简单、可重复的示例吗？我会尽力帮忙的。

org.apache.spark.api.java.function.MapFunction;

baseDs.map(new MapFunction<CompanyInfoRecordNew, CompanyInfoTransformedRecord>() {...}, encoder);

FlatMapFunction<Original, Derived>  flatMapFunction = ( record ) ->{

            List<Derived>  transRecList = new ArrayList<Derived>();

            Derived  transRec = new Derived();

            transRecList.add(transRec);

            Derived  transRec2 = new Derived();
                transRecList.add(transRec2);

            return (Iterator<Derived>) transRecList.iterator();
        };