我可以从ApacheSpark UDF(java)返回一个Tuple2吗?
我需要一个UDF2,它接受两个参数作为输入,对应于String和mllib.linalg.Vector类型的两个Dataframe列,并返回一个Tuple2。这可行吗?如果是,如何注册此udf() 自定义项定义如下:我可以从ApacheSpark UDF(java)返回一个Tuple2吗?,java,apache-spark,apache-spark-sql,user-defined-functions,Java,Apache Spark,Apache Spark Sql,User Defined Functions,我需要一个UDF2,它接受两个参数作为输入,对应于String和mllib.linalg.Vector类型的两个Dataframe列,并返回一个Tuple2。这可行吗?如果是,如何注册此udf() 自定义项定义如下: UDF2<String, org.apache.spark.mllib.linalg.Vector, Tuple2<String, org.apache.spark.mllib.linalg.Vector>> get_item_data =
UDF2<String, org.apache.spark.mllib.linalg.Vector, Tuple2<String, org.apache.spark.mllib.linalg.Vector>> get_item_data =
(String id, org.apache.spark.mllib.linalg.Vector features) -> {
return new Tuple2<>(id, features);
};
import org.apache.spark.sql.types.DataType;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.mllib.linalg.VectorUDT;
List<StructField> fields = new ArrayList<>();
fields.add(DataTypes.createStructField("id", DataTypes.StringType, false));
fields.add(DataTypes.createStructField("features", new VectorUDT(), false));
DataType schema = DataTypes.createStructType(fields);
UDF2获取项目数据=
(字符串id,org.apache.spark.mllib.linalg.Vector features)->{
返回新的Tuple2(id,features);
};
有一个模式
,可定义如下:
UDF2<String, org.apache.spark.mllib.linalg.Vector, Tuple2<String, org.apache.spark.mllib.linalg.Vector>> get_item_data =
(String id, org.apache.spark.mllib.linalg.Vector features) -> {
return new Tuple2<>(id, features);
};
import org.apache.spark.sql.types.DataType;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.mllib.linalg.VectorUDT;
List<StructField> fields = new ArrayList<>();
fields.add(DataTypes.createStructField("id", DataTypes.StringType, false));
fields.add(DataTypes.createStructField("features", new VectorUDT(), false));
DataType schema = DataTypes.createStructType(fields);